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ABSTRACT 


This research effort focused on the use of Advanced Warfighting Experiments 
(AWEs) to support acquisition decisions. Specifically, the thesis evaluated the 
effectiveness of the Army Task Force XXI AWE in providing information to support 
investment decisions and refinement of requirements for information age technologies. 

A detailed analysis of the 1997 Operational Test and Evaluation Command 
(OPTEC) Live Experiment Assessment Report identified program developmental 
recommendations. Data were collected from appropriate program offices and user 
representatives to determine the perceived utility of the recommendations and level of 
implementation. Qualitative data detailing why specific recommendations were or were 
not implemented were used to determine the contributing factors to a program’s ability to 
benefit from participation in the experiment. 

Overall, fifty-two percent of the OPTEC recommendations were reported as either 
fully or mostly implemented. Other potential benefits of AWE participation were 
identified to include (1) marketing and exposure of program, (2) refinement of user 
requirements, and (3) information on integration, interfaces, and interoperabilty. Risks 
from participation in the AWE included (1) a poor return on investment, (2) potential 
negative exposure, and (3) extensive changes in requirements. Recommendations to 
enhance the value of participation in AWEs are included. 
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I. INTRODUCTION 


The principle of science, the definition, almost, is the following: The test 
of all knowledge is experiment. Experiment is the sole judge of scientific 
‘truth.’ But what is the source of knowledge? Where do the laws that are 
to be tested come from? Experiment, itself, helps to produce these laws, 
in the sense that it gives us hints. But also needed is imagination to create 
from these hints the great generalizations—to guess at the wonderful, 
simple, but very strange patterns beneath them all, and then to experiment 
to check again whether we have made the right guess. [Feynman, 1995, 


p2] 
A. BACKGROUND 


From March 1996 through October 1997, the Army conducted the Task Force 
XXIJ Advanced Warfighting Experiment (TF XXI AWE), culminating with a live exercise 
at the National Training Center, during Rotation 97-06, March 1997. The purpose for 
conducting the AWE was to provide sufficient data to validate digitizing the battlefield 
and support credible assessments on which to base future procurement decisions. The TF 
XXI AWE was meant to be a tool for resolving issues and reducing risk early in the 
program development process and determining the adequacy of requirements, design, and 
new system capabilities before committing major resources. A total of ninety-three TF 
XX] new technology initiatives were initially included in the AWE. | 

The stated TF XXI AWE objectives include experimenting with advanced 
technologies and concepts that leverage capabilities of information age technologies and 
providing information to support investment decisions on the most promising initiatives 
[ECC, 1996]. The AWE was also meant to help the U.S. Army Training and Doctrine 
Command (TRADOC) refine requirements and develop solutions for Force XXI. The 
final Live Experiment Assessment Report, prepared by the U.S. Army Operational Test 
and Evaluation Command, included full assessments of most participating TF XXI. 
initiatives. OPTEC provided observations and specific developmental recommendations 
for each initiative. AWE documentation does not directly address the purpose and goals 
of the experimental process as it relates to the acquisition program manager. This thesis 
attempts to partially define goals and objectives for the program manager to consider 
when presented with the opportunity to participate in an AWE. 





B. PURPOSE 


This thesis studies the use of Advanced Warfighting Experiments to support 
material acquisition decisions. Specifically, the thesis evaluates the effectiveness of the 
Army Task Force XXI Advanced Warfighting Experiment (TF XXI AWE) objective of 
- providing information to support investment decisions and refinement of requirements for 
information age technologies. Research includes a detailed analysis of the 1997 
Operational Test and Evaluation Command (OPTEC) Live Experiment Assessment 
Report to identify program developmental recommendations. Data were collected from 
appropriate program offices and user representatives to determine the perceived utility of 
the recommendations and level of implementation. Subjective data detailing why specific 
recommendations were or were not implemented were used to determine the contributing 
factors to a program’s ability to benefit from participation in the experiment. The thesis 
includes recommendations on how to best use Advanced Warfighting Experiments to 


support the acquisition program manager. 
C. SCOPE 


The scope of this thesis is limited in that it concentrates on the 1997 Task Force 
XXI Advanced Warfighting Experiment and no previous or subsequent experiments. 
Only initiatives with detailed recommendations provided by OPTEC were considered. 
The research does not include an analysis of the Joint Venture XXI doctrinal initiatives or 
organizational change initiatives but only the developmental program recommendations. 
This thesis will not determine the advantages or disadvantages of the AWE process 
beyond the support of acquisition decisions. The results of the research do not determine 
the overall success or failure of the TF XXI AWE. However, it concentrates on material 
acquisition issues, the immediate use of OPTEC recommendations for initiative 
development, and the contributing factors behind the use or non-use of those 


recommendations. 


D. | RESEARCH QUESTIONS 


1. What were the objectives of the Army’s Task Force XXI Advanced 
Warfighting Experiment, what was the structure of the experiment, and how was 
information gained from the experiment used in making acquisition decisions? 











2. Were the specific recommendations derived from the Task Force XXI 
Advanced Warfighting Experiment (AWE) used to support investment decisions and to 
refine requirements of participating initiatives? 

3. What were the contributing factors to a program’s ability to benefit from 
participation in the Advanced Warfighting Experiment? 

4. What are the characteristics of programs that are best positioned to gain valued 
investment and requirements information from participation in Advanced Warfighting . 
Experiments? 


E. METHODOLOGY 


The methodology used in this thesis will consist of the following steps. | 

1. Conduct a literature search of books, magazine articles, CD-ROM 
systems, internet and other library information resources. 

2, Conduct a thorough review of the TF XXI AWE Experiment Plan and 
OPTEC Live Experiment Assessment Report. | 


3: Creation of tailored surveys for each program initiative and dissemination 
to each applicable program office and user. 

4. Follow up interviews of pertinent program managers, users, and test 
officials. 

2; Conduct a statistical analysis of collected data regarding implementation 


of initiative recommendations. 
6. Conduct an analysis of collected comments on contributing factors to a 
program’s ability to benefit from participation in the Advanced Warfighting Experiment. 
d. Conduct an analysis and develop a summary description of the 
characteristics of a program that is best positioned to participate in future AWEs. 


F. THESIS OUTLINE 


I. Introduction. This chapter provides information on the scope, 
methodology, and purpose of the thesis. 
| i. AWE Structure and Literature Review. This chapter reviews the 
background and structure of the Army's experimental process and reviews the literature 
addressing experimental research methods. 


I. § Empirical Study Description. This chapter details the design of the survey 
instrument and the analysis strategy. | 

IV. Findings on the Use of Derived Information in Supporting Acquisition 
Decisions. This chapter provides a statistical analysis and conclusions from gathered data 
on the implementation of specific recommendations derived from the AWE. 

V. Analysis of Contributing Factors to Benefiting from Advanced 
Warfighting Experiments. This chapter addresses the subjective comments made by 
survey participants and interviewees and illustrates lessons learned on the contributing 
factors to benefiting from the AWE. It also describes the characteristics of a program that 
is best positioned to gain from participation in future AWEs. | 

VI. Conclusions and Recommendations. This chapter summarizes lessons 
learned from the thesis and makes recommendations to enhance the AWE process to 


support the acquisition manager. 
G. EXPECTED BENEFITS OF THIS THESIS 


This thesis provides the acquisition manager with information necessary to make 
decisions about participation in Advanced Warfighting Experiments. The thesis assists 
acquisition managers in gaining maximum benefit from participation in future 
experiments. The research assists planners in tailoring future experiments to better 


benefit participating material programs. 











Il. AWE STRUCTURE AND LITERATURE REVIEW 
A. THE TASK FORCE XXI ADVANCED WARFIGHTING EXPERIMENT 


iF Background 


In February 1995, the Chief of Staff of the Army approved the Army Force XXI 
concept. The Force XXI concept served to modernize and redesign equipment and 
procedures to meet the emerging twenty-first century threat. Battlefield digitization was 
the central theme for the envisioned Force XXIJ. Digitization includes the linking of all 
weapon platforms and vehicles at the brigade level and below into a tactical internet 
capable of sharing a vast amount of information. Digitization was expected to provide 
improved situational awareness and command and control capabilities. The Army 
designed the Advanced Warfighting Experiment to find the most promising technologies 
to lead the Army to Force XXI. [Lickteig, 1996] 

The Army labeled its effort to modernize to Force XXI as “Joint Venture.” Joint 
Venture was an iterative cycle of concept development, force design, and experimentation 
[Lickteig, 1996]. The Army’s senior leadership described the Joint Venture Campaign 
Plan as “a comprehensive approach to redesign the force—organized around 
information—to be inherently more versatile and flexible” [Lickteig, 1996, p3]. Through 
the initial efforts of Joint Venture, the Army discovered it needed a more pragmatic and 
responsive research method that would clearly provide relevant and credible analysis to 
validate the premise of Force XXI [TRADOC, 1996]. The Army developed the 
Advanced Warfighting Experiments (AWE) as the focal effort in establishing Force XXI. 
The AWEs were based on “an iterative sequence and mix of warfighting simulations— 
live, constructive and virtual—in which soldiers and units conduct realistic tactical 
operations” [Lickteig, 1996, p6]. 

The Task Force XXI AWE was one of a series of AWEs designed to implement 
the Joint Venture plan. The Joint Venture Experimentation Plan is detailed in Figure 1 
and illustrates the path from the current brigade, division, and corps design to the 
modernized Force XXI. The AWE series began with the Focused Dispatch (FD) AWE. 
Focused Dispatch consisted of a heavy battalion task force conducting simulated 


exercises from February 1995 through August 1995. FD AWE focused on developing 
digitized training support packages and validating doctrine and techniques for digitizing 
heavy forces. The FD AWE was followed by the Warrior Focus (WF) AWE. Warrior 
Focus was conducted from May 1995 through November 1995 using a light infantry 
brigade and culminated with a training exercise at the Joint Readiness Training Center 
(JRTC) at Fort Polk, Louisiana. The WF AWE assisted in the Army developing tactics, 
techniques, and procedures (TTPs) in operating and equipping a digitized light infantry 
battalion. The TF XXI AWE was followed by the Division (DIV) AWE held in 
November 1997. The Army designed the DIV XXI AWE to provide insights on echelons 
above division and joint digitized operations. [ECC, 1996] 


. ROLUNG BASELINE 





Figure 1. Joint Venture Experimentation Plan [PM FBCB2, 1995] 


2 Purpose and Goals 


The central hypothesis of the TF XXI AWE was developed by the Army’s 
Training and Doctrine Command and states "that if information-age battle command 
capabilities and connectivity exists across all battlefield operating system (BOS) 
functions in a brigade task force, then increases in lethality, survivability, and tempo will 
be achieved" [ECC, 1996, 1.b.1]. The overall objective of the AWE was to "focus 











modernization efforts on the most promising technologies...that have the greatest 
potential for enhancing force capabilities" [ECC, 1996, 1.c.1]. 

The Experiment Directive published five specific objectives for the AWE: 

a. Assess Force XXI Battle Command Brigade and Below (FBCB2) 
Applique and Tactical Internet capabilities to refine FBCB2 requirements. _ | 

b. Assess digitized brigade combat service support concept. 

c. Experiment with advanced technologies and concepts that se 
capabilities of information age technologies. 

d. Refine digitized tactics, —— and procedures (TTP) for 
brigade operations. 

e. Provide information to support investment decisions on the most 
promising initiatives. [ECC, 1996] 

Fundamental to the Army’s process of Joint Venture is the concept that 
development involves “continuous experimentation, discovery learning, and iterative 
refinement" [Lickteig, 1996, p4]. The rapid advance of technology has resulted in an 
inherent inability for Army leadership to predict the eventual composition and 
organization of its force. Information technology is providing an unprecedented level of 
capability to provide a common picture of the battlefield situation to all participants. The 
defining purpose of the AWE is to provide a forum to implement continued, iterative 
improvements. 

AWE documentation does not act address the purpose and goals of the 
experimental process as it relates to the acquisition program manager. This thesis 
attempts to partially define goals and objectives for the program manager to consider 
when presented with the opportunity to participate in an AWE. 


3. AWE Scope and Methodology 


a. Scope 


The TF XXI AWE initially consisted of thirty-six Joint Venture issues and 
ninety-two new technology initiatives. The Joint Venture issues were proposed research 
questions regarding organizational doctrine, tactics, procedures, leader development, 
organization, and information management. The new technology initiatives included 
advanced technology products in various stages of development [ECC, 1996]. Officials 


reduced the number of participating technology initiatives prior to the experiment based 
on the ability of the initiative to contribute to the overall goals of the experiment. The 
experiment plan focused on the initiatives central to digitization, such as the Applique 
(now titled Force XXI Battle Command Brigade and Below — FBCB2) and those 
initiatives that contributed to the full exercise of these central programs. This thesis 
concentrates on these initiatives and does not conduct an analysis of the value obtained 
from study of the Joint Venture issues. 

The live exercise consisted of platoon, company, battalion, and brigade 
sized units conducting train-up and simulated combat operations against an aggressive 
opposing force. The experiment used pre-digitization exercises to develop a baseline to 
measure the effects of the digital enhancements and the resulting increases in unit 
effectiveness. The TF XXI AWE was designed to measure the holistic effect of 
synchronizing all elements of a combined arms team and to replicate “the same free flow 
operations and decisions that would’be expected on the battlefield” [ECC, 1996, 1.c.8]. 


b. Initiative Selection Procedures 


TRADOC selected initiatives to participate in the TF XXI AWE based on 
their ability to contribute to the objectives of the exercise. Some initiatives were logically 
included because they formed the basic infrastructure of the Army’s overall digitization 
plan, such as Applique/FBCB2 and the Tactical Internet. TRADOC selected other 
systems because of their ability to contribute to the data flow required to exercise the 
digitization backbone, such as the Longbow Apache Helicopter and the Linebacker Air 
Defense Artillery system. Emerging technologies that were new concepts to the Army in 
terms of doctrinal employment and capabilities determination formed a third group, such 
as the Stingray Combat Protection System and the Lightweight Video Reconnaissance 
System (LVRS). Finally, even though they did not directly support the goals of the 
experiment, a fourth group of initiatives were included because of their status as 
emerging technologies. It is unclear how formal and rigorous the selection process was 
for the TF XXI AWE. However, from the results and lessons learned from the AWE, the 
Army has developed a structured system for initiative selection in future AWEs that is 
designed to insure that each system constructively supports the goals and objectives of the 


experiment. 











The formal issue and initiative submission and review process conducted 
by TRADOC for the 1999 Joint Contingency Force AWE includes four phases: (1) 
submission of proposed issues to the TRADOC Analysis Center (TRAC) and the 
appropriate battle lab, (2) consolidation and initial review of issues and initiatives at the 
TRADOC level, (3) approval of issues and initiatives by the selection committee, and (4) 


. the continuous monitoring and refinement of the approved issues and initiatives. The 


main criteria for selection include four components: (1) the initiative’s relationship to an 
AWE hypothesis or objective, (2) the availability of the initiative by the designated 
“Everything-in-Place Date”, (3) the availability of the initiative to be integrated into the 
architecture of the AWE, and (4) the ability of the submitting agency to bear the complete 
costs associated with participation. The submitting agency must also explain the impact 
of the issue or initiative on the Army’s DOTMPL framework. The DOTMPL framework 
is a structured method the Army uses to discuss the effects of new concepts in terms 
doctrine, organization, training, materials, people, and leadership. [TRADOC JCW- 
AWE, 1998] 


c. Data Collection 


During the 1997 TF XXI AWE, automated instrumentation collected 
digital data while audio recording equipment was used to capture voice message traffic on 
FM radio nets. Real time casualty assessment (RTCA) instrumentation collected data on 
force effectiveness and movement rates. Simulated Area Weapons Effects-Multiple 
Integrated Laser Engagement system (SAWE-MILESII) provided results of unit 
engagements. Most significantly, Subject Matter Experts (SMEs) provided direct input to 
qualitatively measure initiative contribution to force effectiveness. The use of SMEs was 
essential in understanding the complex interactions between initiatives [OPTEC, 1997]. 
The primary emphasis for the experiment were those initiatives that directly contributed 
to situational awareness improvements. Most other technological initiatives were 
considered of secondary importance and. were not measured quantitatively. These 


individual system contributions were not analyzed individually but as part of the total 


digitization package [ECC, 1996]. 


d. Analysis Plan 


Training and Doctrine Command Analysis Center - White Sands Missile 
Range (TRAC-WSMR), the Operational Evaluation Command (OEC), and the Test and 
Experimentation Command (TEXCOM) jointly developed the analysis plan for the TF 
XXI AWE. The analysis plan detailed issues, measures of effectiveness (MOEs), and 
analysis techniques to be used during the live simulation [ECC, 1996]. TEXCOM’s 
representative for the live portion of the experiment was the Army’s Operational Test and | 
Evaluation Command (OPTEC). The analysis team, combined with the Army Battle 
Labs, invested a significant effort to complete a set of coordinated, finalized initiative 
assessment plans [OPTEC, 1997]. 


4, Limitations 


The planning documentation for the TF XXI AWE acknowledges the inherent 
limitations to the Army’s study. “The planned TF XXI experiments do not follow the 
formal study process, the traditional test and evaluation process, nor do they follow the 
strict procedures of a laboratory experiment” [ECC, 1996, 1.c.16]. The attempt to 
combine training with assessment of digitization does not allow a controlled, repeatable 
experiment. Because of the training focus of the exercise, data collection efforts must be 
unobtrusive. This requirement limits the control of factors and conditions required to 
support analysis [ECC, 1996]. | 


a. Repeatability 


The free-play nature of the experiment limited the ability to repeat the 
results of any particular scenario. The sheer number of variables makes this tasks 
infeasible. Data collection failures due to instrumentation or equipment malfunctions 
were identified risk areas that could not be mitigated. 


b. Data Collection 


Data collection on all systems besides the Applique, tactical internet, and 
electronic interfaces was limited to SME observations and user surveys. The experiment 
directive spoke directly to the data collection limitations. “The majority of the initiatives 
will not have objective data to support milestone decisions. The SME and user data on 
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these second priority initiatives will be most useful in refining requirement documents 
and highlighting potential strengths and weaknesses” [ECC, 1996, 1.c.16]. Lessons 
learned from previous AWEs demonstrated to the TF XXI planners that SME insights 
and observations were the most meaningful in terms of reliability and significance. 
“Attempting to isolate the individual contributions of each experimental system to overall 
unit performance is impossible” [ECC, 1996, 2.b.1.a]. 

The non-intrusive data collection required of an experiment overlaid on 
training significantly limited the ability to gather meaningful information. Data collection 
over its entire process lifecycle from conception to termination requires a resource 
commitment that is infeasible in a free-play environment. The effects of specific actions 
and their ramifications on other variables could not adequately be measured with the 
limitations of the experiment [OPTEC, 1997]. 


c. Interactions 


The large number of interacting initiatives participating in the AWE 
combined with uncontrolled and random variations lead to hopelessly confounded results 
that could not be isolated to demonstrate respective contributions. In true laboratory 
experiments, variables are modified individually or according to specific patterns that 
provide an ability to decipher impacts of variation. The nature of an experiment overlaid 
upon a training exercise prevents the direct control required to limit corruption of data by 
variable interaction. The large effects of interactions cause the subjective SME 
observations to represent a significant level of importance. 


d. Small Sample Size 


The relatively small sample of focused iterations during the live 
experiment coupled with the large number of initiatives involved provide for statistically 
insignificant results. Sufficient time and resources are not available during the AWE to 
develop statistically meaningful data. This small sample sizes make it very difficult to 
“prove” hypotheses as true. [ECC, 1996]. The RAND corporation has years of 
experience conducting research at the National Training Center and has significanrit 
expertise in conducting effective quantitative and qualitative experiments. The National 
Training Center conducts monthly "rotations" lasting 14 days, training a new brigade- 
sized unit each month. The TF XXI AWE consisted of one single NTC rotation. RAND 


I] 


admits that the single-rotation AWE concept is limited in what quantitative results it can 
provide. RAND has found that at least twelve rotations’ worth of data is required to 
provide statistically meaningful results. RAND does go on to state that AWEs can 
readily provide significant qualitative results. [Grossman, 1995]. 


e. System Immaturity 


As stated in the experiment directive, some new systems participating in 
the experiment were immature, surrogate in nature, and did not have the capability to 
demonstrate actual or desired performance [ECC, 1996]. 


5. Results 


Official press releases after completion of the AWE labeled the exercise nearly a 
complete success. The commanding officer of TRADOC, General William W. Hartzog, 
stated that about eighty-five percent of the prototypes performed to requirements. Of the 
rest, ten percent needed improvements to be useful and five percent were “ideas whose 
times have not yet come” [TRADOC PAO, 1997]. 

The primary feedback mechanism for the AWE was Operational Test and 
Evaluation Command’s Live Assessment Report. The report provided summations for 
most participating initiatives as well as specific recommendations for improvements in 
technology. cginouay: the assessment report summarized the achievements of the 
AWE. 

a. . Better definition and understanding of future requirements. 
b. Early identification of systems having potentially high payoff 
capabilities, as well as systems needing significant developmental work. 


C. Independent assessment to the Army’s senior leadership. 
d. Early feedback to program managers. 
e. Acceleration of the normal acquisition process by placing 


prototype systems into the hands of soldiers. [OPTEC, 1997] 
| In its final report, OPTEC also identified significant lessons learned from the 
AWE process. While OPTEC recognized that the AWE offered excellent opportunities 
for the identification and refinement of system requirements, the combination of 
experiment and training does not support operational testing requirements. In order to 
gain beneficial test data, training effectiveness would be reduced. However, OPTEC did 
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acknowledge that the AWE was an excellent opportunity to identify critical operational 
issues and criteria (COIC) that would be beneficial for use in future formal tests and 
evaluations [OPTEC, 1997]. | 


B. DEFINING THE PROCESS: TRAINING, TESTING, ASSESSMENT, 
DEMONSTRATION, OR EXPERIMENT 


1. Overview 


The Army leadership has spent considerable effort in defining the task and 
purpose of their experimentation. Planners initially chose the word “experiment” to 
differentiate the exercise from two different categories of activities that had already been 
labeled “test” and “demonstration.” A presentation for a past AWE stated that leadership 
should “reinforce experiment versus demonstration versus test” [Lickteig, 1996]. The 
Advanced Warfighting Experiment had a central hypothesis and attempted to apply a 
rudimentary scientific method but was criticized, as described in the next section of this 
thesis, as not being a true scientific study. 

Concepts and terms used in the development of an experiment or evaluation plan 
often have a “richness of meaning” [Babbie, 1990, p119]. That is, they contain a variety 
of elements that attempt to summarize complex phenomenon. To permit rigorous 
research, such general concepts must be reduced to specific, empirical indicators. These 
indicators must be reduced to simplified, precisely defined terms. In the case where 
concepts cannot be precisely defined, the outcomes cannot be measured precisely. A 
tenet of scientific research is that data are not collected but rather created. The creation of 
data is highly dependent on the definition of variables and attributes [Babbie, 1993]. | 

The AWE experiment directive used terms such as “gathering information”, 
“making decisions”, “gaining insight”, “evaluation”, “validation” and “assessment” 
without clearly defining the application of the terms. It is unclear how the Army initially 
applied the term experiment and how experimentation would interact with the other 
developmental activities already in place. The AWE analysis plan also identified that 
there would be a wide range of analytical rigor in the assessment of initiatives. A final 
complication introduced by the designers of the AWE was the overlay of the event on a 
pre-established training exercise at the National Training Center (NTC), Fort Irwin, 
California. The NTC is commonly referred to as the Army’s most highly rated training 
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opportunity for mechanized forces. The description of the AWE as a training event 
additionally confounded the purpose of the exercise. | 

Through the experience gained in the TF XXI AWE, following exercises, and 
detailed studies of the Army’s experimentation process, the Army has more clearly 
defined its experimental method, the terms used in its analysis plans, and the interaction 
experimentation plays with other developmental activities. The Army has determined 
that future exercises must be based on sound, integrated, all-inclusive analysis 


methodology. 
2. Training 


The Army defines training as “the instruction of personnel to individually and 
collectively increase their capability to perform specific functions or tasks” [CALL, 
1999]. OPTEC stated in the TF XXI AWE Live Simulation Assessment Plan that “the 
primary objective of the exercise is training” [ECC, 1996, Annex C, Appendix B, para. 
2.3.1]. The Army’s National Training Center was founded with the purpose of providing 
the most realistic training exercises possible short of actual war. Secondarily, the NTC 
was designed to provide “lessons learned” for the Army. 

The NTCs training mission emphasis has a significant implication on its ability to 
serve aS an environment for an Advanced Warfighting Experiment. Although NTC 
rotations approximate actual combat, they do not exactly replicate it. Even with extensive 
improvements in technological capability, the environment of the NTC is designed such 
that it is unlikely that the evaluation force will achieve victory in any one engagement. 
The NTC also focuses on certain missions and battlefield operating systems. Those 
initiatives not realistically exercised in the training environment should not be included in 
the experiment. The situation has added complexity in that training is combined with the 
introduction of new equipment. It is difficult to separate problems related to training and 
problems related to new initiatives. Actual effects of applied initiatives may also be 
limited due to the need to continue the exercise in a proper force correlation to serve the 


training objective. [Grossman, 1995] 
3. Testing 


Testing in the Army is divided between developmental and operational activities. 
Both are designed to be assessments of an item's ability to meet designed requirements 
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and parameters. Operational testing is an official requirement to conduct independent 
analysis of equipment prior to production. While the live simulation assessment plan 
included in the experiment directive a dendritic of functional issues, measures of 
effectiveness (MOE), measures of performance (MOP), and data sources as normally seen 
in an operational test and evaluation plan, representatives from the Army’s Test and 
Evaluation Command were adamant that the exercise was not a “test” and would not 
replace proper operational test and evaluation of any initiative. 

The live simulation assessment plan stated that the planned Task Force XXI 
experiments would not follow the traditional operational test and evaluation process. 
Because of the severe limitations imposed by the nature of the experiment combined with 
the needs of the training exercise, any data collected would be insufficient in drawing any 
conclusions required for a.true operational test [ECC, 1996]. The NTC is not designed as 
a test and evaluation center. Although the NTC observer/controllers are extremely 
beneficial in the conduct of AWEs, they will not compromise their training function 
[Grossman, 1995]. In a personal interview, a representative from the Headquarters of 
OPTEC stated that while the AWE did not count as operational testing for any 
developmental item, it was included in the spectrum of “continuous evaluation.” He went 
on to state that developmental and operational testing are merging and that if proper 
actions are taken by the developmental community after the AWE, that the system might 
require less testing and scrutiny when the time for operational testing arrives. 


4. Assessment 


The Army defines assessment as “the determination of the overall effectiveness” 
[CALL, 1999]. The term “assessment” was not clearly defined in the TF XXI experiment 
directive. However, future AWE frameworks clearly define the term “assess” as “to 
analyze critically and judge definitely, the nature and merit of mature systems, 
technologies, and procedures” [TRADOC JCF-AWE, 1998, A-1]. The analytical rigor of 
the term “assess” is considered very high. 

Assessment plans developed for the TF XXI AWE were a result of “a collective 
effort between the proponent and OPTEC” [ECC, 1996, Appendix B, Annex 1]. 
Initiative proponents provided draft assessment plans to OPTEC. OPTEC revised the 
plans to match the available collection assets and also added issues of their own. 
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Proponents for initiatives were normally the specific Battle Lab that had responsibility for 


that particular area of interest 
5. Demonstrations 


The term “demonstration” was clearly defined for experiments following the TF 
XXI AWE as a hierarchical experimental term that recognizes the level of maturity of a 
system.. “Demonstrate & Deploy” was defined as “to show by virtual, simulated, or 
actual operation a potential capability for prototype systems; technologies, and 
procedures” [TRADOC JCF-AWE, 1998, A-1]. The analytical rigor for demonstrations 
is considered high. : 

The Army conducts two different demonstration programs, Advanced Technology 
Demonstrations (ATDs) and Advanced:Concept Technology Demonstrations (ACTDs). 
The primary objective of the ACTD process is to “accelerate and facilitate application of 
mature advanced technologies to solve important military problems” [USAMRMC, 1997, 
pl]. ACTDs conduct realistic and extensive military exercises to provide the user an 
opportunity to evaluate utility and gain experience with mature technologies. ACTDs are 
funded by participating initiatives, supplemented by funding from the ACTD funding 
line. ATDs demonstrate the potential for enhanced operational capability and help to 
speed the maturation of advanced technologies. They are funded by initiative Research, 
Development, Test, and Evaluation (RDT&E) funds. [USAMRMC, 1997] 

The need to properly label the activities that would become Advanced 
Warfighting Experiments as something separate from demonstrations was evident early in 
the program development process. General Gordon Sullivan, the architect for Joint 
Venture stated “first we called them demonstrations, and that didn’t fly because we knew 
if it was a demonstration, that wouldn’t be satisfactory for any of us. It would be an 


experiment” [Lickteig, 1996]. 
6. Experiments 


Experiment connotes discovery learning. Experiments are “formative exercises to 
see what works and what doesn’t” [Lickteig, 1996, p15]. The strict application of the 
scientific method to experimentation is a precisely controlled research method. The 
original term of Advanced Warfighting Demonstrations was changed to Advanced 
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Warfighting Experiments to show the application of more realistic conditions that tasked 
soldiers and equipment to wartime levels. [Lickteig, 1996] 

The Army’s Advanced Warfighting Experiment appears to be based on a strictly 
scientific model. The Army has developed hypotheses and emphasized the validation of 
models against baselines. The Army has also stated in briefings that “the scientific 
method paradigm must be adapted” [Lichteig, 1996, p8]. 

The U.S. Army Research Institute recommended that the Army implement a 
formative evaluation method that focuses on exploration, explanation, and improvement. 
The Army has developed an analytic method it refers to as Model-Experiment-Model- 
Validate (MEMV). Models are initially developed by battle labs and experimented 
through simulation. The models are then adjusted based on derived experimental data 
_and validated through live exercises. The role of the AWE in the Army’s experimental 
model is discussed in the following section. [Lickteig, 1996] 

Advanced Warfighting Experiments are not the Army’s only experimental 
_ activity. The Army’s Battle Labs also conduct their own warfighting experiments. The 
Battle Lab Warfighting Experiments (BLWE) are smaller, more focused assessments of a 
single battle dynamic. [Lickteig, 1996] 


7. Other Key Definitions 


a. Validate 


The term validate has been defined for future AWEs to mean “confirm” or 
“substantiate”. “Tt is not used in the strict experimental sense where there is a comparison 
of alternatives” [TRADOC, 1996, 2.e]. 


b. Gain Insights 


The term “gain insights” has been defined for future AWEs as an 
experimental term that recognizes the level of maturity for a concept. The term “gain 
insights” is defined “to obtain a clear and immediate understanding of emerging concepts, 
systems, technologies, and procedures” [TRADOC JCF-AWE, 1998, A-1]. 
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c. Objectives 


Obj ectives are specific, operational statements detailing the desired 
accomplishments of a program. The planning and implementation of programs are 
predicated on the need to reduce the gap between objectives and reality. Effectively 
stated objectives use strong verbs, state only one purpose or aim, specify a single end- 
product or result, and specify the expected time for achievement. [Rossi, 1993] 


C. EXPERIMENTAL RESEARCH METHODS 


“Scientific research has two primary goals: description and explanation. 
Researchers measure the empirical distribution of values on variables (description) and 
the associations between variables for purposes of explaining the distribution of values” 
[Rossi, 1993]. A wide spectrum exists on the application of a strict scientific model on 
warfighting experiments. The U.S. Army Research Institute believes that research 
objectives should determine research methods [Lickteig, 1996]. The complex nature of 
the AWE may not be facilitated by a strict experimental model. Basic research theory 
demonstrates that proving:a true difference among two states is much more difficult than 
proving no difference [Lickteig, 1996]. AWEs will not benefit from lessons learned in a 
“no difference found” environment. The Army has admitted that it is “experimenting 


about experimenting” [Lickteig, 1996, p15]. 
1. The AWE Application of the Scientific Method 


The scientific method of experimentation is a precisely controlled research 
method. Scientists experiment by randomly assigning participants to an experimental and 
control group and strictly control factors “extraneous to the causal variable of interest” 
[Lickteig, 1996, p15]. Multiple studies have criticized the Army’s use of the scientific 
method in experimentation and recommended more structured and applicable methods. 
The AWEs have neither random assignment of participants nor strict control over 
variables. For the TF XXI AWE, the experimental and control group were the same, 
equipped differently. The wide range of variability in the exercise fuels debate on the 
validity of any of the conclusions reached. 
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According to a study done by the U.S. Army Research Institute, the Army has 
recognized the limitations in its implementation of the scientific method but may not have 


adequately addressed them. 


Rather than empirical control over conditional differences, or a systematic 
design to reduce variation, the plan proposes to document such differences 
(e.g., issues, design, structure, equipment) and conditions unique to each 
exercise, in a relational data base. The documentation of all exercise 
conditions is essential for interpretation of findings. However, it is no 
substitute for the controls required to establish commensurate conditions, 
systematic changes in treatment, and definitive or validated outcome 
improvements. [Lickteig, 1996, p16] 


The Research Institute proposes that AWEs do not prevent designers from 
controlling strict controls on variables. The necessary control might be achieved by 
dedicating a portion of the experiment to small, detailed sub-experiments. The results of 
the scientific study should also focus on providing solutions rather then identifying 
failure. [Lickteig, 1996] 


2. Formative Versus Summative Evaluations 


The U.S. Army Research Institute published its findings on appropriate research 
methods for Advanced Warfighting Experiments and believes that the overarching 
research strategy should consist of a balance between formative and summative 
evaluations [Lickteig, 1996]. Formative and summative evaluations differ in their focus 
and role. 

Formative studies are defined as “evaluative activities undertaken during the 
design and pretesting of programs to guide the design process [Rossi, 1993, p104]. These 
formative studies may be simple or “as complex and comprehensive as full-blown 
evaluations” [Rossi & Freeman, 1993]. Formative evaluations focus on intermediate 
goals and play a productive role in their achievement. Formative evaluations are 
conducted in developmental stages to help form or improve the system for the user 
[Lickteig, 1996]. 

Summative evaluations address final goals as opposed to the intermediate goals of 
formative evaluations. Summative evaluations address the success or failure of goal 
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achievement as opposed to taking an active roll in their achievement. Summative 
evaluations generally provide information for an external audience. [Lickteig, 1996] 
While both formative and summative evaluations can be conducted during AWEs, 
an emphasis on formative issues may avoid a complete conclusion of failure driven by 
summative evaluations. The macro-level focus of AWEs can better be served by the 
formative evaluation model. Summative evaluations require exacting methods that 
cannot be facilitated in the free-play, training environment of the AWE. The Joint 
Venture concept of iterative, continual improvement also models formative evaluations 
more directly. Since Joint Venture is formative in nature, the belief that AWEs cannot 
satisfy the requirements for operational test and evaluation is supported due to the 
summative nature required for operational testing. Finally, the premature focus on 
summative results of individual initiatives might lead to the faulty conclusion that 
advanced technology devices are not beneficial to the force as a whole. [Lickteig, 1996] 


3. The Credible Uses Framework 


In 1998, the RAND Corporation published its findings on the application of the 
“Credible Uses” (CU) framework to Advanced Warfighting Experiments [Lucas, Banks, 
Vye, 1998]. | | 

The CU framework, originally documented by James A. Dewar (et al.) [Dewar, et 
al.1996], is based on a decision-to-experiments ladder (DEL) that directly links 
experiments to decisions [Lucas, Moore, Vye, 1998]. Advanced Warfighting 
Experiments are limited in their ability to predict real world outcomes. Experimental data 
comes from single or few unrepeatable events. Safety restrictions, lack of realism, and 
unknown composition of future threats make AWE’s “weakly predictive” at best. 
Analysis of these “weakly predictive” models cannot produce quantitatively significant 
data. Dewar’s study of weakly predictive models demonstrated how the CU framework 
could be used to adjudicate clearly defined hypothesis. Lucas has stated that significant 
up-front analysis is required to maximize experimental results from an AWE. Decisions 
on important issues require an objective and traceable link from experimental results to 
decisions. [Lucas, Banks, Vye, 1998] 

The decision-to-experiment ladder is clearly founded in the scientific method. 
With it, experiments are designed to affect decisions and those decisions are based on 
experiment outcomes. The decision-to-experiment ladder includes: issues, decisions to 
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be made, argument to support the decisions, hypotheses to be adjudicated, experiments to 
resolve hypotheses, and analysis and measures to implement decisions [Lucas, Banks, 
Vye, 1998] 


a. Issues 


While using the weakly predictive model presented in AWEs, the primary 
issues to be addressed must be strictly limited. How many issues, and to what extent they 
may be examined, will be defended by the successful completion of the decision-to- 
experiment ladder of the CU framework. [Lucas, Banks, Vye, 1998] 


b. Decisions to be Made 


The CU process includes specific decisions that can be resolved by an 
analytical process, specifically by experimentation. Potential decisions made within the 
decision to experiment ladder must explicitly stated. Specificity in stating issues is 
critical to experimental design. Decisions must be stated precisely enough so that 
experiments can be properly designed to test them. [Lucas, Moore, Vye, 1998] 


c. Argument to Support the Decisions 


When using the CU framework, users must identify tentative arguments 
that are needed to support the decisions to be made [Lucas, Moore, Vye, 1998]. By 
referencing how the issues relate to the decisions to be made, analysts know exactly the 
strength of the argument needed to make the decision. Where strong prior belief exists in 
the decision to be made, very little experimental evidence may be required to make the 
decision. Where substantial evidence is required, it may not be possible to resolve the 
decision given the limited resources of the experiment. Arguments must be “sufficient 
and succinct. Superfluous arguments lead to added complexity and the identification of 
unmanageable amounts of experiments" [Lucas, Moore, Vye, 1998, p8]. This 
identification of arguments to support decisions is critical in the design of the experiment 
and is much more productive than conducting the experiment and then deducing what can 
be inferred from the evidence. [Lucas, Banks, Vye, 1998] | 
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d. Hypotheses to be Adjudicated 


In the CU framework, AWEs must be designed to test specific hypotheses 
that support credible arguments. By definition, a hypothesis is “an assumption used as a 
basis for an argument or investigation or a theory that explains a set of facts and can be 
tested by further investigation” [Lucas, Banks, Vye, 1998, p18]. According to Lucas, the 
hypotheses used in the TF XXI AWE were faulty in that they were either obviously true 
or were not specific enough to be subject to experimental test. [Lucas, Banks, Vye, 1998] 


e. Experiments to Resolve Hypotheses 


The experiment used to resolve the hypotheses must include what things 
will be varied and what data will be extracted. The experiments in the AWE process can 
be live, virtual, or constructive, as required to satisfy the needs of the hypothesis. To 
maximize the analysis capability from the experiment, significant up-front analysis must 
be completed. Care must be given in choosing the exact variables and scenarios for the 
experiment. It might be determined that the requirements of the hypothesis are not 
feasible within the given constraints of the exercise. Experiments should be as objective 
as possible. Subjective data can produce erroneous conclusions. According to the study 
by Lucas, the Army has historically been effected by subtle biases that have prevented its 
adaptation to new technologies [Lucas, Moore, Vye, 1998]. An analysis of the 
experiment and its assumptions is key to the eventual credible use of the resulting data. 
[Lucas, Banks, Vye, 1998] 


f. Analysis and Measures to Implement Decisions 


For the experiment to be used credibly, it must be designed to support 
decisions. Because of the limited data gathered from AWEs, there must be an “objective 
and traceable link from experimental results to decisions on important issues” [Lucas, 
Banks, Vye, 1998, p16]. The procedures and products of the experiment must be 
explicitly stated to ensure consistency, traceability, and credibility. [Lucas, Banks, Vye, 
1998] 
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D. UTILIZATION OF EVALUATIONS 


The Advanced Warfighting Experiments are evaluations of effectiveness and 
efficiency of combat operations. No matter the complexity or integrity of the scientific 
process of the exercise, “in the end, the worth of evaluations must be judged by their 
utility” [Rossi, 1993]. To achieve maximum benefit from experiments, the actual 
evaluation must be tailored to the specific program, the stage of activity in the program, 
and the needs of the stakeholders in the program. The evaluation of innovative programs, 
such as the digitization of the Army, may require leadership to rethink some aspects of 
the program, including the program objectives, characteristics, and outcome measures. 
[Rossi, 1993]. 

The ability of an organization to achieve its objectives is measured by three 
criteria: (1) the extent that measurable objectives have been specified, (2) the plausibility 
and testability of assumptions that link the application of resources to the program 
activities, and (3) the level at which those in charge of the program have the motivation, 
ability, and authority to manage and implement change. Research has shown that the 
results of evaluations might not be used to refine and modify programs because 
leadership may resist, be uncooperative, or fail to grasp the purpose of the studies. Often, 
while programs may have a number of objectives, they may not be sufficiently well 
defined to be managed to desired endstates, may not have valid assumptions, or may be 
invalidated by management’s inability to implement change. [Rossi, 1993]. 

The way that evaluations are used can be classified in three categories. First, 
direct or instrumental use of evaluations means that the specific findings and 
recommendations are directly documented and implemented. Second, conceptual 
utilization refers to the general use of evaluations to influence thinking about issues. 
Conceptual utilizations do not lead to the adoption of specific programs or policies but 
provide evidence that supports general methods and approaches. Finally, persuasive 
utilization applies findings to support or refute beliefs or to attack the status quo.- [Rossi, 
1993]. 

Five factors affect the utilization of evaluation findings: (1) relevance to the 
problem, (2) communication between researchers and users, (3) information processing 
by users, (4) plausibility of research results, and (5) user involvement or advocacy. Also, 
leadership will be influenced by the perceived truthfulness of the evaluation. 
Management will measure truthfulness by determining if the research was conducted by 
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proper scientific methods and if the results are compatible with their experience, 
knowledge, and values. [Rossi, 1993]. | 

The extent that the data obtained from the TF XXI AWE was utilized by the Army 
program managers involved will be the central point of investigation in this theses. 


E. CHAPTER CONCLUSIONS 


A review of the goals, objectives, and limitations of the TF XXI AWE overlaid on ~ 
a background study of the role of experiments in the Army, the various analyses and 
recommendations of experimental research methods, and the theoretical utilization 
characteristics of evaluations, raises questions about the possible utility of the information 
derived from the TF XXI AWE for the program manager. This background information 
will be considered and applied to the results of the objective and subjective study that 
follows. : 

One product from the TF XXI AWE was a performance analysis of the 
participating new technology initiatives. OPTEC provided specific recommendations for 
each initiative in its Live Experiment Assessment Report. These recommendations 
detailed changes to user requirements, desired technology improvements, and integration 
issues. This thesis studies the utility of the AWE to program managers, as discussed by 
Rossi, by investigating the levels at which these recommendations were implemented and 
the factors affecting utilization including relevance to the problem, communication 
between participants, information processing by users, the plausibility of research results, 
and user involvement [Rossi, 1993]. The thesis also investigates the extent to which the 
AWE was a formative evaluation by measuring its impact on the programs it was directed 


to improve. 
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II. EMPIRICAL STUDY DESCRIPTION 
A. RESEARCH AND APPROACH 


1. Research Focus 


The thesis research focused on gathering ratings from program managers and user 
representatives as to the degree of implementation of the Operational Test and Evaluation 
Command’s specific initiative recommendations relative to their specific program. These 
recommendations were derived from the Live Experiment Assessment Report for the TF 
XXI AWE. Secondly, a research survey was conducted on the perceived validity of the 
recommendations and to identify contributing factors towards a program’s ability to 
benefit from participating in the Advanced Warfighting Experiment. | 


2. Research Approach 


Research began with a comprehensive review of literature addressing the goals, 
purpose, techniques, and results of the Army’s experimental process. A detailed review 
of the TF XXI AWE Experiment Plan and OPTEC’s Live Experiment Assessment Report 
contributed to the understanding of the Army’s experimental program. Background 
research included general research and evaluation methods and the utilization of 
evaluations. Surveys were tailored for each program initiative and disseminated to each 
applicable program office and user representative. Research included follow-up 
interviews of pertinent program managers and users. Analysis was then conducted to 


draw conclusions relevant to the research focus. 


B. SAMPLING 


1. Included Initiatives 


This research does not include an analysis of the Joint Venture XXI doctrinal 
initiatives or organizational change initiatives but only the new technology developmental 
programs participating in the TF XXI AWE. Of the ninety-two emerging technology 
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initiatives included in the experiment, thirty-six initiatives were included in the research. 
Certain initiatives were excluded from the research for the following reasons: (1) the 
initiatives was not evaluated by OPTEC during the AWE, (2) OPTEC did not provide any 
substantive recommendations by which to measure implementation, (3) initiative was a 
doctrinal or organizational change with no material program, (4) program was 
subsequently terminated after the AWE and no representatives could be found to provide 
input towards the research. Of the thirty-six initiatives selected, surveys were issued to 
thirty-four. One initiative was subsequently excluded because no willing participants 


could be found. 
2. Selection of Respondents 


Seventy-two respondents were identified to participate in the survey. First, a 
representative from the responsible program management office was identified for each 
of the thirty-six initiatives. Program managers were selected based on their direct 
experience with the AWE and their ability to provide programmatic insights into the 
process and results of the AWE. Secondly, a user representative was identified, in most 
cases the combat developer responsible for the program. Of the seventy-two identified 


offices, surveys were administered to sixty-seven. 


C. SURVEY DESIGN AND INSTRUMENT 


1. Survey Design 


The survey was designed to collect both objective and subjective data. The 
objective survey items were designed to collect data on the specific level of 
implementation of recommendations made in the Live Experiment Assessment Report. 
Subjective items were included to gather perceptions and opinions from specific program 
offices and user representatives. The subjective items were structured from the RAND 
Corporation’s application of the “Credible Uses” (CU) framework to Advanced 
Warfighting Experiments as explained in detail in Chapter II of this thesis [Lucas, Banks, 
Vye, 1998]. 

The CU framework, originally documented by James A. Dewar (et al.) [Dewar, et 
al.], is based on a decision-to-experiments ladder (DEL) that directly links experiments to 
decisions [Lucas, Moore, Vye]. Advanced Warfighting Experiments are limited in their 
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ability to predict real world outcomes, since experimental data generally comes from 
single or few unrepeatable events. Safety restrictions, lack of realism, and unknown 
composition of future threats make AWE’s “weakly predictive” at best. Analysis of these 
“weakly predictive” models cannot produce quantitatively significant data. However, 
Dewar’s study of weakly predictive models demonstrated how the CU framework could 
be used to adjudicate clearly defined hypothesis [Lucas, Banks, Vye]. The items in the 
subjective portion of the survey are framed on the decision-to-experiment ladder to allow 
participants to demonstrate through their responses the extent to which this decision 
ladder was implemented during the AWE. 


2. Survey Instrument 


The survey instrument utilized a combination of ordinal measurements reflecting 
the level of implementation achieved for each OPTEC recommendation and factors 
affecting implementation, as well as open-ended questions designed to gain subjective 
perceptions about the AWE process. The survey contained four sections. 


a. Section One 


Section one asked respondents to rate the level of implementation of the 
specific recommendations made for their program initiative by the Army Operational Test 
and Evaluation Command's Live Experiment Assessment Report. Each survey was 
tailored to include only those specific recommendations applicable to the initiative in 
question. The complete list of specific OPTEC recommendations is included as 
Appendix A. Participants were asked to indicate the term that best described the extent 
that the recommendation was implemented: FULLY, MOSTLY, LIMITED, NOT AT 
ALL. Participants were also asked to provide narrative comments explaining the factors 
that influenced the degree of implementation. 


b. Section Two 


Section two contained questions about the included program's experiences 
in the 1997 TF AWE. Participants were asked to rate, using a five-point scale (1=high, 
5=low), questions that were derived from the decision-to-experiment ladder [Lucas, 
Moore, Vye, 1998]. The questions covered all of the applicable components from issue 
development to analysis and implementation. Respondents were given the opportunity to 
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give a brief narrative explanation of each rating. In addition to the objective ratings, 
open-ended questions were included (see questions one, six, and seven in Figure 2) to 
gather contributing factors and characteristic description of programs best positioned to 
benefit from AWEs. The specific questions contained in Section Two are listed in Figure 
2. 


Part 2: 


The following questions gather information on your program’s experience in the 1997 TF XX] 
Advanced Warfighting Experiment. In the area given, please provide your perceptions and 
opinions. Where applicable, please circle the appropriate rating that best answers the given 
question. Please provide as much detailed information as possible. You may use the back of 
this form or attach additional pages. 


1. What were the specific developmental ISSUES being addressed on your program at the time 
of its participation in the AWE and what DECISIONS were to be made from gathered data? 


Explain: 


2. To what extent were you able to tailor or influence your program’s specific activities in 
the AWE to relate to the issues and decisions you faced as an acquisition manager? 


igh) [1] [2] [3] [4] [5] (Low) 
Explain: 


3. How valuable were the data and recommendations gained from participation in the AWE 
in making decisions as an acquisition manager? Explain. 


Gigh) 1) 2) £4) - (4 [5] Gow 
Explain: 


4. To what extent did your program benefit from participating in the AWE? How? 
(High) [1] [2] [3] [4] [5] (Low) 
Explain: 


6. What were the contributing factors to your program’s ability to benefit from participation in 
the AWE? 


Explain: 


7. Based on your program’s experience in the 1997 AWE, describe the characteristics of a 
program that would best be situated to benefit from participation in a future AWE. 


Explain: 
Figure 2. Section Two of Survey Instrument 
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Cc. Section Three 


Section three gathered respondent opinions on reasons why 
recommendations made by the 1997 Advanced Warfighting Experiment were or were not © 
fully implemented. The questionnaire attempted to include an exhaustive list of potential 
impediments or positive factors for implementation. Participants were asked to rate each 
of the items on a scale of one to five. A rating of one signified a highly significant factor 
on the degree of implementation. A rating of five signified that an item was NOT an 
influencing factor. Participants were also asked to place a mark next to the most critical 
factor. Section three of the survey is shown as Figure 3. 


Part 3: 


The following section gathers your opinion on reasons why recommendations made by the 1997 
Advanced Warfighting Experiment were or were not fully implemented. Please rate each of the. 
items below on a scale of 1 to 5. A rating of 1 signifies a highly significant factor on the degree of 
implementation. A rating of 5 signifies that an item was NOT an influencing factor. Please place 
a * next to the most critical factor. The first set of items are possible impediments to 
implementation. The second are possible positive factors for implementation. 


Reasons for less than full implementation. Highly - Not 


Significant Significant 
1. Lack of money to implement recommendations from AWE. [1}) [2] Bl] [4] [5] 
2. Lack of time to implement recommendations from AWE. [1] [2] [3) [4] [5] 
3. Test data and recommendations from AWE were not valid. 11) [2] B [4] [5] 
4. AWE evaluators did not understand user’s requirements. [1] [2] £3] | [4] [5] 
5. AWE data and recommendations were not accurate. 1] (2) {3) [4] [5] 
6. AWE recommendations were not technically feasible. 1) —) @&) [4 ff) 
Factors supporting implementation. Highly - Not 

Significant Significant 
1. Issue was considered high priority by users. (1] [2] [3) [4) [5] 
2. Issue was considered high priority by program office. [1] [2] [3] [4] [5] 
3. Issue had strong political backing. 11) 2] 8) 4 5] 
4. Recommended actions were fully funded. 1) [2] [B) [4 [5] 


Figure 3. Section Three of Survey Instrument 
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d. Section Four 


In section four, respondents were asked to provide any additional 
comments on their program’s participation in the AWE including any information that 
might assist acquisition managers in gaining maximum benefit from participation in 
future experiments or that would assist planners in tailoring future experiments to better 


benefit participating programs. 
D. DATA COLLECTION 


First, potential respondents from the selected offices were contacted by email to 
explain the survey and its purpose. The thesis topic and potential derived benefits were 
explained. Potential respondents were asked if they had both the necessary background 
experience in the TF XXI AWE and the estimated time of one hour to complete the 
survey. Surveys were then administered to.participants by issuance and submission of 
Microsoft Word documents via electronic mail. 

Interviews were conducted in person with seven representatives from the Army’s 
Night Vision Reconnaissance Surveillance and Target Acquisition Office at Fort Belvoir, 
Virginia and two representatives from the Program Management Office for Bradley 
Systems at the Army's Tank, Automotive, and Armaments Command (TACOM) in 
Detroit, Michigan. Interviews were conducted face-to-face during a combined one week 
on-site visit from August 24 - 27, 1999. These interviews were designed to solicit more 
detailed information addressing the same questions included in the administered survey. 
A sample of questions used in the interviews is shown in Figure 4. 

All respondents to the written survey and personal interviews were informed that 
the research would be conducted on a non-attribution basis and that participants would 
remain anonymous. It was believed that anonymous input would result in a more candid 


disclosure of information. 
E. LIMITATIONS OF RESEARCH DESIGN 


The following is a description of the major factors identified prior to the survey 


administration that could limit the study. 
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1. As a material developer and acquisition manager, what is your overall perception or 
impression of the AWE process? 


2. What kind of specific feedback did you receive on your program’s performance in the 
AWE? Was it valid? Could you take advantage and utilize the feedback. 


3. Has your program undergone any major changes since the AWE? 

4. How did the maturity of your program effect its performance at the AWE? 
5. Were you able to tailor the analysis plan for your program with OPTEC? 
6. How did your program benefit from participating in the AWE? 


7. What were the risks involved with participating in the AWE? How were those risks 
mitigated? 


8. What amount of effort was involved in participation in terms of resources, time, 
manpower, or dollars? How does the cost of participation compare to the benefits 
received? 


9. What role did politics play in outcomes and perceptions of programs? 
10. To what extent was the AWE a marketing effort for PMs? 


11. How would you characterize a program that would stand to significantly benefit from 
participation? ; 


- Figure 4. Sample Interview Questions 


1. Scope 


The research was limited to programs participating in the TF XXI AWE and no 
other Army experiments. The available programs may not represent the full breadth of 
possible outcomes of the AWE process. Subsequent Army AWEs may have been refined 

' to invalidate any conclusions resulting from this study. The sample of included programs 
might not be representative and may limit the relevance and validity of conclusions. 

Some program offices have been disbanded subsequent to the TF XXI AWE. 
Representatives for certain initiatives were not available to contribute to the study. High 
levels of personnel turnover in Army agencies might lead to the conclusion that 
respondents did not have relevant experience to evaluate the AWE. In some cases, details 
of the planning process and results of the AWE were not documented and were therefore 
unavailable for the study. 
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2. Reliability 


Reliability is “a matter of whether a particular technique, applied repeatedly to the 
same object, would yield the same result each time” [Babbie, 1990, p132]. Although 
every effort was made to conduct the research in a rigorous manner, the study relied on 
data from multiple sources with unknown biases. All data were based on opinions and 
perceptions that were not independently verified. Depending on the amount of data 
actually collected and agreement across the sample, levels of confidence can be measured 


to qualify conclusions. 
3. Validity 


Validity is “the extent to which an empirical measure adequately reflects the real 
meaning of the concept under consideration” [Babbie, 1990, p133]. The inclusion level 
participating AWE initiatives studied is sufficient to create valid conclusions on the 
outcomes of the TF XXI AWE. However, any derived conclusions might not be 
applicable to programs participating in future AWEs due to changes in the acquisition 


process since 1997. 
F. RESPONSE STATISTICS 


Sixty-seven surveys were administered to both program managers and user 
representatives of thirty-five different AWE initiatives. A total of thirty-eight 
respondents returned completed surveys for a response rate of 56.7%. Only twenty-eight 
of the thirty-five originally included initiatives had survey responses. The remaining 
seven initiatives were subsequently excluded from the thesis. Of the twenty-eight 
included initiatives, only ten had responses from both the program manager and user 
representative. While the cumulative level of response was adequate to reach general 
conclusions on the acquisition use of derived AWE data, there was an inadequate level of 
response to compare program manager and user representative input or to measure if any 
significant differences existed between the two perspectives. 

Both program management and user representative offices participated in the 
survey. Completed surveys were received from nineteen program management offices 
and nineteen user representative offices. The program management office surveys were 
completed by program, project, or product managers. The users were represented by the 
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appropriate Directorate of Combat Developments (DCD) or other offices to include 
TRADOC System Managers (TSMs) and Subject Matter Experts (SMEs) from branch 
specific doctrine offices. Respondents included military, Department of Defense (DoD) 
civilians, and support contractors. Military respondents ranged from Major (O-4) to 
Colonel (O-6) and had an average of 1.9 years experience on the program in question. 
Civilian respondents ranged from GS-12 to GM-15 and had an average of 7.5 years 
experience on their program. The minimum amount of time any respondent had with | 
their program was 12 months. Also included were four civilian contractors who directly 
supported program offices or user agencies. 

Data were collected on the maturity of programs at the time of participation in the 
AWE. Respondents were asked to indicate their "phase of development" at the AWE. 
Responses were then transcribed into five categories to be used in analysis. Program 
maturity is summarized in Table 1. 


Table 1. Reported Maturity of Programs at the AWE 








Concept Exploration (CE) or Advanced Technology 
Demonstration (ATD) 


Program Definition and Risk Reduction (PDRR) ae ae 


Engineering and Manufacturing Development (EMD) 


Production (LRIP) 
Production, Full Rate Production (FRP), or Full Scale 
Production (FSP) 


Overall, given the scope, reliability, and validity of the response demographics, 
the data are adequately representative of the TF XXI AWE participating program 
managers and user representatives. The collected data are sufficient to reach findings on 





the use of AWEs to support acquisition decisions. 
G. ANALYSIS STRATEGY 


The thesis survey asked respondents to subjectively rate recommendation 
implementation and perceptions about the AWE using a number scale. For this analysis, 
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the provided numeric ratings were inverted (S=high; 1=low). Arithmetic means, medians, 
and modes were computed for each question to determine relationships and effects. 
Mean is defined as the average and was computed by dividing the sum of the terms by the 
number of the terms. The median term is defined as that value of an ordered set below. 
and above which there are an equal number of values. Mode is defined as the most 
frequent value of a set of data. A confidence interval was computed that should include 
the true value of the parameter 95% of the time. The confidence interval was computed 
as plus or minus 95% of the area under a standard normal curve multiplied by the value of 
the standard deviation and divided by the square root of the sample size. For most 
analysis, medians were used for comparison so that outliers would not have a significant 
impact on results. 

| Median results for the various collected factors are compared against one another 
and analyzed to determine trends and overall effects on the ability to benefit from the 
AWE. The Kruskal-Wallis method is used to determine a P value, or level of equality, of 
the various factor medians. The Kruskal-Wallis method is computed on multiple samples 
as a nonparametric alternative to one-way analysis of variance. A nonparametric analysis 
implies that there is no assumption of a specific distribution for the population. 

The Kruskal-Wallis hypotheses are: HO: the population medians are all equal 
versus H1: the medians are not all equal. Kruskal-Wallis H is calculated on the basis of 
sums of ranks for combined groups. Data from all samples are ordered, letting n be the 
number of observations in any particular sample and N is the number of observations in 
all samples combined. The data are renumbered from 1 to N, with 1 corresponding to the 
lowest score. Using these rank scores, data for each sample are listed by rank. These 
rank scores are added up for each sample, and the sums are the r scores in the formula 
below. Also, let k be the number of samples. The Kruskal-Wallis H is computed as: H = 
I2/(N(WN + 1))* SUM(r’7/n) - 3(N + 1) and degrees of freedom = k - 1. H is distributed 
approximately as chi-square. H is then used in a chi-square table with (k - 1) degrees of 
freedom. If the critical value of chi-square for the desired significance level is equal to or 
less than the computed H value, then the researcher rejects the null hypothesis that the 
samples do not differ on the criterion variable. In the analysis that follows, the 
significance level for each Kruskal-Wallis test is given. [Garson, 1999] 

The planned method of analysis included a review of the respondent surveys to 


identify the following: 
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6. 





The average level of implementation of all included initiative 
recommendations. 
The overall perception of the programs ability to tailor or influence the 
initiatives specific activities in the AWE. 
The overall perceived value of the data and recommendations gained from 
participation in the AWE as acquisition managers. 
The overall perceived benefit gained from participating in the AWE. 
If recommendations reflected trends related to 
- level of program maturity 
- extent to which program was able to tailor or influence the initiative’s 
activities inthe AWE 
- perceived value of the data and recommendations gained 
- the perceived benefit received from the AWE 
A hierarchy of reasons why recommendations made were or were not fully 


implemented. 


Finally, the collected subjective comments were analyzed to draw conclusions to 


answer the primary and secondary research questions and develop a characteristic 


description of programs that are best positioned to gain valued investment and 


requirements information from participation in Advanced Warfighting Experiments. 
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IV. FINDINGS ON THE USE OF DERIVED INFORMATION IN SUPPORTING 
ACQUISITION DECISIONS 


A. INTRODUCTION 


This chapter presents the summaries of the subjective ratings gathered from the 

thesis survey. Section A of this chapter discusses the statistical measures gathered from 
the survey to include: | 

| e the level that OPTEC recommendations were implemented 

e the extent to which managers could tailor or influence their program’s 

participation in the AWE 

e the perceived value of recommendations gained from the AWE 

e the level that programs benefited from participation 

e the factors influencing implementation or lack of implementation 

Section B provides an analysis of recommendation implementation based on the 
above program issues. Section C summarizes this chapter. 


B. FINDINGS 


1. Implementation of Recommendations 


Respondents were asked to indicate the term that best described the extent that 
specific recommendations made by the OPTEC Live Experiment Assessment Report 
were implemented by their respective programs. The survey corresponding with each 
program included from two to seven recommendations derived from the Assessment 
Report. The total number of recommendations evaluated was one hundred. Possible 
responses included "Fully", "Mostly", "Limited", or "Not at All." For analysis purposes, 
the response choices were given an ordinal value such that the number four represented 
"Fully" and the number one represented "Not at All." The included survey instruments 
provided one hundred total recommendations to be evaluated. Where differences existed 
between the perceived level of implementation as reported by the program manager and 
that reported by the user representative, an average score was used. 
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The average level of reported implementation was 2.52, translating into a rating 
between "Limited" and "Mostly". The median response was 3.0, signifying "Mostly" 
implemented. A histogram of reported levels of implementation is shown as Figure 5. 
Of all the included recommendations, 52% were either fully or mostly implemented. It is 
important to note that there was a fairly even split between those recommendations that 
were fully or mostly implemented and those with limited or no implementation. 
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Figure 5. Reported Levels of Recommendation Implementation 
2. Ability to Tailor or Influence a Program’s Activities in the AWE 


Respondents were asked to rate to what extent they were able to tailor or influence 
their program’s specific activities in the AWE to relate to the issues and decisions they 
faced as acquisition managers. Examples of possible tailoring included modifications of 
scenarios to insure data availability for specific issues, influence over tactics and 
techniques used by participants in the experiment, and contribution to the data collection 
or analysis plan. For the purposes of analysis in this thesis, the gathered data were 
converted to a numerical scale from one to five with five being a high ability to tailor or 
influence and one being a low ability. A total of twenty-eight responses were evaluated. 
Ten respondents chose not to answer the question. 

‘The average, level of ability to tailor or influence activities was 2.14 on a scale of 
one to five, translating into a rating slightly lower than the middle of the range. The 
median response was 1.0, signifying a very low ability to tailor their activities. The mode 
of responses was also 1.0. A histogram of reported levels of implementation is shown as 
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Figure 6. It is important to note that 54% of all respondents reported a very low ability to 
tailor or influence activities. 
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Figure 6. Reported Ability to Tailor or Influence a Program’s Activities in the AWE 


3. Perceived Value of Data and Recommendations Gained from 
Participation in AWE 


Respondents were asked to rate the value of the data and recommendations gained 
from participating in the AWE in making decisions as acquisition managers. For the 
purposes of analysis, the gathered data was converted to a numerical scale from one to 
five with five signifying that the data was of high value and one signifying low value. A 
total of thirty-one responses were evaluated. Seven respondents chose not to answer the 
question. 

The average perceived value of the data and recommendations from the AWE was 
3.10 on a scale of one to five, translating into a rating directly midpoint of the range. ‘The 
median response was also 3.0. The responses had no significant mode. A histogram of 
reported perceived values is shown as Figure 7. The collected data indicate that 42% of 
the included programs considered the value of the data and recommendations gained as 
highly valuable (rated 4 or 5) while 39% perceived the value as very low (rated 1 or 2). 
The degree of perceived value varied significantly among initiatives. | 
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Figure 7. Reported Value of Data and Recommendations Gained from AWE 


4. Perceived Benefit from Participating in AWE 


Respondents were asked to rate to what extent their program benefited from 
participating in the AWE. For the purposes of analysis, the gathered data were converted 
to a numerical scale from one to five with five signifying the AWE was highly beneficial 
and one signifying that it was less beneficial. A total of thirty-four responses were 
evaluated. Four respondents chose not to answer the question. | 

The average perceived extent of benefiting from the AWE was 3.21 on a scale of 
one to five, translating into a rating slightly above the midpoint of the range. The median 
response was also 3.0. The responses were bimodal at 3.0 and 5.0, signifying a varying 
degree of benefiting. A histogram of reported perceived values is shown as Figure 8. 
The data indicates that 45% of included programs benefited from the AWE (rated 4 or 5). 
However, 33% of programs reported little to no benefit (rated 1 or 2). 


5. Factors Affecting the Implementation of Recommendations 


Respondents were asked to rate the extent that specific factors supported the 
implementation of AWE recommendations and also the extent that specific factors 
represented reasons why recommendations were not fully implemented. The collected 
data was converted to a numerical scale from one to five with five indicating that the 
factor was highly significant and one indicating that the item was not a significant factor. 
Six items were presented to the respondents as reasons for less than full implementation. 
A total of thirty-six responses were evaluated for reasons for less than full 
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implementation. Two respondents chose not to rate any factors. Jour items were 
presented to the respondents as factors supporting implementation. A total of thirty-four 
responses supporting implementation were evaluated. Four respondents chose not to rate 
any factors supporting implementation. 
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Figure 8. Reported Extent that Program Benefited from Participation in the AWE 
a. Reasons for Less Than Full Implementation 


As shown in Figure 5, seventy-five percent of the one hundred included 
recommendations were not fully implemented. A summary of the medians and associated 
95% confidence intervals for the factors inhibiting full implementation are illustrated in 
Figure 9. According to the survey, a lack of money was the most significant reason that 
recommendations derived from participation in the AWE were not fully implemented. 
This factor had a median rating of 4.0 with a rating of five meaning the item was highly 
significant. This factor rated far above any other item, including the second most cited 
reason, that test data and recommendations were not valid. The least significant item, 
according to respondents, was that the recommendations were not technically feasible. 
Other items, to include a lack of time to implement, AWE evaluators not understanding 
the user’s requirements, and data and recommendations not being accurate, were not 
reported as significant factors. 
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Figure 9. Median Values of Reasons for Less Than Full Implementation 





b. Factors Supporting Implementation 


As shown in Figure 5, fifty-two percent of the one hundred included 
recommendations were either fully or mostly implemented. The most significant factor 
supporting implementation of AWE recommendations was that the recommendation was 
considered a high priority by the user. This factor had a median value of 4.0 on a scale 
from one to five. This factor was higher than any other rated item, including that the 
issue was fully funded, was considered a high priority by the program office, or had a 
strong political backing. A summary of the factor medians and associated 95% 
confidence intervals for the thirty-four included initiatives are graphically illustrated in 


Figure 10. 
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Figure 10. Median Values of Factors Supporting Implementation 


C. ANALYSIS 


1. The Effects of Perceived Value of Data and Recommendations on 
Implementation 


_ The perceived value of AWE data and recommendations was compared to the 
levels of recommendation implementation. Survey responses were grouped according to 
reported perceived value of data. Figure 11 illustrates the median level of 
recommendation implementation by the reported value of data and the associated 95% 
confidence intervals. Those reporting very low value in the received AWE data indicated 
a median implementation level of 2.0 on a scale of one to five with 95% confidence that 
the level of implementation was between 1.0 and 3.5. Those reporting very high value in 
the received AWE data indicated a median implementation level of 2.75 with 95% 
confidence that the level of implementation was between 1.0 and 3.9. Those initiatives 
reporting high values of data value reported slightly higher rates of recommendation 
implementation. However, the wide range of responses and resulting large confidence 
intervals suggests that the extent that recommendations were implemented varied only 
slightly across the entire reported spectrum of perceived value of data. 


43 


A Kruskal-Wallis H test was conducted to determine the equality of the sample 
medians. The test resulted in a P of .757 showing no significant difference among the 
median ratings of the value of AWE data. Therefore, the data indicates that the perceived 
value of AWE data and recommendations was not in correlation with the level of 


recommendation implementation. 
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Figure 11. Median Levels of Recommendation Implementation by Value of Data 





To determine if the perceived value of data had a causative effect on the level of 
recommendation implementation, a comparison was made of the reported value of data 
and implementation levels. Implementation levels were grouped into two samples: (1) 
fully and mostly, and (2) limited and not at all. Figure 12 illustrates the median level of 
value of data by the reported implementation level and the associated 95% confidence 
intervals. Those programs with recommendations that were fully or mostly implemented 
reported a median value of data equal to 3.0. Those programs with recommendations that 
had limited implementation, or no implementation at all, reported a median value of data 
equal to 2.0. A trend is evident that suggests programs with high levels of 
recommendation implementation reported higher levels of value of data. | 

A Kruskal-Wallis H test was conducted to determine the equality of the sample 
medians. The test resulted in a P of .322 signifying that the two levels of implementation 
are not significantly different in terms of the value of the data received from the AWE. 
Although a trend is evident that suggests programs with high levels of recommendation 
implementation reported higher levels of value of data, the small sample size and large 








variance leads to a statistically insignificant conclusion. An analysis of the respondent 
comments helps to explain the suggested trend and variance in responses. 
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Figure 12. Median Levels of Value of Data by Recommendation Implementation 


A program manager for a conceptually new communication program stated that 
the feedback from the AWE was very valuable in the decision making process because it 
was based on actual user feedback and very specific user requirements. However, the 
recommendations provided by the AWE Live Assessment Report were not all technically 
feasible. The program’s technology was also considered commercial-of-the-shelf (COTS) 
and did not lend itself to recommendation implementation. The user representative from 
a program currently in full production stated that the data gained was highly valuable in 
developing tactics, techniques, and procedures. However, the recommendations requiring 
contractor developmental effort could not be implemented because of a lack of available 
research, development, and procurement funding as well as the inability to be included in 
production models due to the late stage of maturity for the initiative. The program 
manager of the same initiative stated that the value of the data was only moderately 
valuable in identifying "connectivity issues" (the interoperability of digital systems) 
associated with TF XXI and that most of the recommendations fell outside of the 
responsibility of the program management office. 

Data indicated that the OPTEC recommendations were either fully or mostly 
implemented by one program in the concept exploration phase of development. 
However, the program manager of this initiative stated that the value of the AWE data 
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was low because those in leadership positions were not willing to acknowledge the 
demonstrated capabilities of the system. Another program manager stated that the 
recommendations had low value in making decisions because of the statistically 
insignificant data provided by an unrepeatable experiment. 

It can be concluded from the above data and statements that the perceived value of 
the data may not have been a significant factor in the implementation of the AWE 
recommendations. The trend pattern of medians suggests a positive correlation. 
However, the small sample size in each group leads to a lack of statistical significance. 
Qualitative comments elaborated the rating results that factors other than perceived value 
of data effected the recommendation implementation. Specifically, participants 
considered data valuable in making decisions as acquisition managers when the 
recommendations met one of three requirements. First, the data was valuable when it 
provided actual user feedback on specific user requirements. Second, data was 
considered valuable when it contributed to the development of tactics, techniques, and 
procedures. Finally, the data was considered valuable when it was provided at a time 


when it could be instrumental in refining requirements and design. 


2. The Effects of Perceived Benefit from Participating in the AWE on — 
Implementation 


The perceived benefit from participating in the AWE was compared to the 
recommendation implementation levels. An acquisition manager’s perception of the 
extent that a program benefited from participating in the AWE was reflected in the level 
that AWE recommendations were implemented. Survey responses were grouped 
according to reported perceived benefit gained from the AWE. Figure 13 illustrates the 
median level of recommendation implementation by the extent that a program benefited 
from the AWE and the associated 95% confidence intervals. The programs that reported 
a very low level of benefit from participating in the AWE indicated a median 
implementation level of 1.0 (not at all) on a scale of one to five with a 95% confidence 
that the implementation was between 1.0 and 2.0. Those reporting a very high level of 
benefit indicated a median implementation of 2.8 (mostly implemented) with a 
confidence interval from 1.5 to 3.0. 

A Kruskal-Wallis H test was conducted to determine the equality of the sample 
medians. The test resulted in a P of .036 showing that the median levels of 
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implementation were significantly different when comparing levels of reported perceived 
benefit. Therefore, the data indicates that the perceived benefit of the AWE was in 
correlation with the level of recommendation implementation. The data indicates that 
those programs receiving moderate to high benefit from the AWE were more likely to 
implement the OPTEC recommendations then those reporting low or very low. 
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Figure 13. Median Levels of Recommendation Implementation by Perceived Benefit 





To determine if the level of recommendation implementation had a causative 
effect on the perceived benefit from AWE participation, a comparison was made of the 
reported perceived benefit and implementation levels. Implementation levels were 
grouped into two samples: (1) fully and mostly, and (2) limited and not at all. Figure 14 
illustrates the median level of perceived benefit by the reported implementation level and 
the associated 95% confidence intervals. Those programs with recommendations that 
were fully or mostly implemented reported a median perceived benefit equal to 4.0. 
Those programs with recommendations that had limited implementation, or no 
implementation at all, reported a median perceived benefit of 3.0. A trend is evident that 
suggests programs with high levels of recommendation implementation reported higher 
levels perceived benefit. 

A Kruskal-Wallis H test was conducted to determine the equality of the sample 
medians. The test resulted in a P of .117 signifying the difference in the two 
implementations levels are not significantly different in terms of the perceived benefit of 
the AWE. However, this difference comes close to the critical value of p < .10 often used 
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for small sample tests. The small sample size and variance leads to a statistically 
inconclusive finding. However, the data shows that their is some effect. An analysis of 
the respondent comments helps to explain the suggested trend and variance in responses. 
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Figure 14. Median Levels of Perceived Benefit by Recommendation Implementation 


A program manager directing a pre-milestone II initiative stated that the AWE 
provided tremendous insight for Program Definition and Risk Reduction (PDRR). All of 
the associated AWE recommendations for this initiative were fully implemented. It is — 
interesting to note that the mean implementation level in those programs receiving the 
most benefit from the AWE (high = 5) was lower than those reporting moderate of 
moderately-high levels of benefit (3-4). The mean recommendation implementation level 
of these programs receiving high benefit was reduced by two specific initiatives. Both 
initiatives reported a high level of benefit received with no implementation of 
recommendations (not at all = 1). The first program stated that none of the 
recommendations fit any of the user’s requirements. The user representative for the 
program stated that the high level of benefit was attributed to the attention and exposure 
that the initiative’s performance generated at the AWE. The second program was a 
logistics and support system that, according to the program manager, was not adequately 
stressed in the AWE environment. None of the recommendations were implemented 
because it was felt that the data and recommendations were not valid or accurate. The 
program manager stated that the focus of the AWE was on combat systems and not on the 
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associated support systems. The main benefit this program manager reported was critical 
feedback from the user on required functionality. | 

The high level of implementation for those programs reporting moderate benefit 
(3) from the AWE was effected mostly by one initiative that reported full implementation 
of all AWE recommendations. The program reported, however, that all but one of the 
shortcomings was identified prior to the AWE and that the AWE was not instrumental in 
identifying these problems. Therefore, the program manager reported high levels of 
recommendation implementation but a moderate benefit from the AWE. 

It can be shown from the collected data and comments that the level of 
recommendation implementation and the perceived level of benefit from the AWE were 
generally related. It cannot be concluded, however, that the implementation of AWE 
recommendations was solely responsible for a respondents perception of benefit, as 
illustrated by the provided comments and slightly lower recommendation implementation 
levels for those reporting a high amount of perceived benefit. A wide range of 
confounding factors effected an initiatives ability to benefit from the AWE. These 
contributing factors to a program’s ability to benefit from participation in the AWE are 
discussed in Chapter V. 


3. Effects of Tailoring a Program’s Activities in the AWE on 
Implementation, Value of Data, and Perceived Benefit 


The reported ability to tailor or influence a program’s activities in the AWE was 
compared to collected data on recommendation implementation. Tailoring was defined 
as the ability to influence the program’s activities to relate to the issues and decisions 
facing the program. Additionally, comparisons were made between the ability to tailor 
and perceived value of data as well as the ability to tailor and the reported level of 
benefiting from AWE participation. For the following comparisons, survey responses 
were grouped according to reported ability to tailor or influence program activities in the 

AWE. 


a. Tailoring and Implementation 


Figure 15 illustrates the median level of recommendation implementation by the 
extent that a program was able to tailor and the associated 95% confidence intervals. The 
median implementation level reported for those programs with a low ability to tailor or 
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_ influence was 2.0 (limited implementation) on a scale of one to five with a 95% 
confidence that the value was between 1.0 and 3.0. The median implementation level 
reported for those programs with a high ability to tailor their activities was 3.5 (mostly to 
fully implemented) with an associated confidence level between 2.6 and 3.9. 
Implementation levels increased as levels of tailoring increased for every factor but one. . 
The low median implementation level for the tailoring factor of four can be attributed to a 
significantly low sample consisting of only one initiative. This initiative was a 
production item that lacked developmental funds to implement the derived 


recommendations. 
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Figure 15. Median Levels of Recommendation Implementation by Level of Tailoring 





A Kruskal-Wallis H test was conducted to determine the equality of the sample 
' medians. The test resulted in a P of .062. Given a significance criterion of p < .10, this 
demonstrates that the levels of tailoring differ significantly in terms of the degree of 
implementation of recommendations. Therefore, the data indicate that the level of 
recommendation implementation level was in correlation with the reported ability to 
tailor. The collected data suggest that as a program manager’s ability to tailor activities in 
the AWE to relate to the program’s issues and decisions increases, so does the 
implementation level of derived recommendations 

To determine if the level of tailoring had a causative effect on the level of 
recommendation implementation, a comparison was made of the reported ability to tailor 
and recommendation implementation levels. Implementation levels were grouped into 


50 











two samples: (1) fully and mostly, and (2) limited and not at all. Figure 16 illustrates the 
median level of tailoring by the reported implementation level and the associated 95% 
confidence intervals. Those programs with recommendations that were fully or mostly 
implemented reported a median perceived benefit equal to 2.0. Those programs with 
recommendations that had limited implementation, or no implementation at all, reported a 
median perceived benefit of 1.0. A trend is evident that suggests programs with high 
levels of recommendation implementation reported higher levels ability to tailor. 
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Figure 16. Median Levels of Tailoring by Recommendation Implementation 


A Kruskal-Wallis H test was conducted to determine the equality of the sample 
medians. The test resulted in a P of .091. Given a significance criterion of p < .10, this 
analysis shows the higher implementation group reported significantly more tailoring than 
the low implementation group. 


b. Tailoring and Value of Data 


Figure 17 illustrates the median level of perceived data value by the extent that a 
program was able to tailor and the associated 95% confidence intervals. The median 
value of data and recommendations reported for those programs with a low ability to 
tailor or influence was 3.0 on a scale of one to five. The median value of data reported 
for those programs with a high ability to tailor their activities was 5.0. The very low 
sample size prevented the calculation of confidence intervals for some levels of tailoring. 
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Figure 17. Median Levels of Perceived Value by Level of Tailoring 





A Kruskal-Wallis H test was conducted to determine the equality of the sample 
medians. The test resulted in a P of .042 demonstrates that the levels of tailoring differ 
significantly in terms of the rated value of data resulting from the AWE. A trend is 
evident in the data that suggests as a program manager’s ability to tailor activities in the 
AWE increases, so increases the perceived value of the data and recommendations in 
making acquisition management decisions. However, the wide range of responses and 
associated confidence intervals requires further analysis of provided comments to reach a 
conclusion on the correlation of tailoring ability with value of data. This concept is 
explored further in Chapter V. 


c. Tailoring and Perceived Benefit 


Figure 18 illustrates the median level of perceived benefit by the extent that a 
program was able to tailor activities in the AWE and the associated 95% confidence 
intervals. The median perceived benefit reported for those programs with a low ability to 
tailor or influence was 3.0 on a scale of one to five. The median value reported for 
perceived benefit of those programs with a high ability to tailor their activities was 5.0. 
The very low sample size prevented the calculation of confidence intervals for some 


levels of perceived benefit. 
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Figure 18. Median Levels Perceived Benefit by Level of Tailoring Ability 





A Kruskal-Wallis H test was conducted to determine the equality of the 
sample medians. The test resulted in a P of .054 demonstrates that the tailoring levels 
differed significantly in their reported benefits of the AWE. The data indicate that a high 
ability to tailor or influence activities in the AWE had a correlation with the perceived 
benefit gained. Those initiatives reporting a low ability to tailor, generally had low levels 
of perceived benefit. 


d. Conclusions on Ability to Tailor 


It can be concluded that the extent that a manager was able to tailor or 
influence a program’s specific activities in the AWE to relate to the program’s acquisition 
issues and decisions directly contributed to the level of recommendation implementation, 
perceived value of data, and the extent that the program benefited from participating. 
However, the wide range of responses and associated confidence intervals requires further 
analysis of provided comments to reach a conclusion on the effects of increased tailoring 
_ ability. These trends are explored in more detail in Chapter V. 
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4. The Effects of Program Maturity on Implementation, Value of Data, 
and Perceived Benefit 


The program’s level of maturity while at the AWE was compared to the data 
collected on recommendation implementation. Additionally, comparisons were made 
between maturity and perceived value of the data and recommendations, and also 
between maturity and the perceived benefit from AWE participation. 

Programs were categorized into maturity groupings based on the standard | 
Department of Defense acquisition management process as detailed in DoD Regulation 
5000.2-R. Phase 0 is Concept Exploration (CE) and typically consists of competitive, 
parallel short-term concept studies. Programs that were Advanced Technology 
Demonstrations (ATDs) are also included in this group. Phase I is Program Definition 
and Risk Reduction (PDRR). In PDRR, programs become defined as one or more 
concepts, design approaches, and/or parallel technologies. Phase II is Engineering 
Manufacturing and Development (EMD). In EMD, the most promising design 
approaches are translated into stable, producible, supportable, and cost-effective designs. 
Also manufacturing and production processes are developed and system capabilities are 
demonstrated through testing. Programs late in EMD begin Low Rate Initial Production 
(LRIP). The initiatives that have begun LRIP have been group under Limited Product 
(LP). Phase III is production, fielding, deployment, and operational support and have 
been grouped as production items (PROD). [DoD 5000.2-R]. 


a. Maturity and Implementation 


Figure 19 illustrates the median levels of recommendation implementation 
by program maturity and the associated 95% confidence intervals for the medians. This 
comparison was conducted using multiple recommendations for each of the thirty-seven 
included programs, resulting in a sample size of one hundred. The highest level of 
recommendation implementation was seen on those programs in Engineering and 
Manufacturing Development (EMD) with a median implementation level of 4.0 (fully 
implemented) on a scale of one to five with a 95% confidence interval from 3.0 to 4.0. 
The lowest levels of implementation were on either end of the spectrum, with Concept 
Exploration (CE) and Advanced Technology Demonstration (ATD) initiatives reporting a 
median of 2.5 and full rate production items having a mean of 2.0. 
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Figure 19. Median Levels of Implementation by Maturity 





A Kruskal-Wallis H test was conducted to determine the equality of the 
sample medians. The test resulted in a P of .021. This finding shows that the maturity 
levels differed significantly in terms of the degree of implementation of 
recommendations. The data indicates a trend in which programs with high levels of 
recommendation implementation were most likely midpoint in programmatic 


development. 
b. Maturity and Value of Data 


Figure 20 illustrates the median level of perceived value of data and 
recommendations by initiative maturity and 95% confidence levels for each median value 
for each of the thirty-seven included programs. The highest median score for value of 
data was reported by those programs in the PDRR phase of development, with a median 
of 5.0 on a scale of one to five with a 95% confidence that the value was between 1.0 to 
5.0. Programs in EMD had the second highest median at 4.0. Programs in Limited 
Production (LP) and Production placed the lowest value on the data received, rating at 2.0 
and 3.0 respectively. Programs in CE also scored low, with a median of 3.5. 

A Kruskal-Wallis H test was conducted to determine the equality of the 
sample medians. The test resulted in a P of .458 indicating no significant difference in 
the maturity levels of programs and their respective ratings of data value. Therefore, the 
data indicates that value of data and recommendations was not in significant correlation 
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with level of maturity. However, the data indicates a possible trend between 
programmatic maturity and the perceived value of the data and recommendations gained 
from the AWE in making decisions as acquisition managers. Those programs in the mid- 
developmental phases of acquisition were more likely to report high perceived value of 
data and recommendations than those programs early or late in development. Further 
analysis of provided comments is required to determine the correlation between these two 
factors. This correlation is explored further in Chapter V. 
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c. Maturity and Perceived Benefit 


Figure 21 details median levels of reported benefit by program phase of 
development with 95% confidence intervals on the medians. The highest reported benefit 
was seen by those programs in PDRR and EMD with a reported median value of 4.0 on a 
scale of one to five with a 95% confidence that the value was between 2.0 and 5.0 for 
PDRR and between 3.0 and 5.0 for EMD. Initiatives in production reported similar 
perceived benefits. Initiatives in CE and limited production had median values of 3.0. 

A Kruskal-Wallis H test was conducted to determine the equality of the 

sample medians. The test resulted in a P of .235 signifying no difference in the median 
reported perceived benefit by levels of maturity. Therefore, the data indicates that 
perceived benefit was not in correlation with level of maturity. However, there is a trend, 
not supported by statistical findings, that those programs in or near the mid- 
developmental phases of acquisition were more likely to report higher levels of benefit 
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then those programs in other phases. Of additional importance, those programs late in 
production reported high levels of benefit. A full analysis of the perceived benefits of 
AWE participation, as reported by acquisition managers is included in Chapter V and 
fully explains these trends 
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Figure 21. Median Perceived Benefit by Maturity 
d. Conclusions on the Effects of Maturity 


It can be concluded that a program’s level of maturity related to the level of 
recommendation implementation. A trend is evident that more maturity did not directly 
transcribe into a higher ability to benefit, but rather points existed on the maturity 
spectrum where a program was best positioned to benefit from the experiment. 
Additionally, a similar pattern of relationship seams to exist between a program’s level of 
maturity and the perceived value of data. The preceding data indicates that a program 
reported the highest level of implementation, perceived value of data, and benefit if it was 
in the mid-development phases at the time of the AWE. Programs early in development, 
or into production, generally reported lower values for these items. However, initiatives 
in production also reported high levels of benefit, even though they did not fully 
implement the AWE recommendations. This suggests that factors other than 
recommendation implementation may have effected a program’s ability to benefit. 
Maturity’s effect on a program’s ability to benefit from participating in the AWE, as well 
as other factors, are explored in Chapter V. 
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D. CHAPTER SUMMARY 


This chapter presents the summaries of the subjective ratings gathered from the 
thesis survey and the findings on the use of information derived from the AWE in 
supporting acquisition decisions. Gathered subjective comments are provided to help 
interpret the data. Overall, programs reported that fifty-two percent of the OPTEC 
recommendations from the AWE Live Assessment Report were either fully or mostly 
implemented. Thirty percent of recommendations were not implemented at all. A 
combined fifty-four percent of initiatives reported a low ability to tailor their activities in 
the AWE to relate to issues and decisions they faced as acquisition managers. 
Respondents indicated only a moderate benefit from participating in the AWE and that 
the data and recommendations received were only somewhat valuable. The most cited 
reason for recommendations not being implemented was a lack of funding. The 
recommendation was most likely to be implemented if it was a high priority of the user. 

The quality and amount of collected data enables one to reach several conclusions 
about a program’s participation in the AWE. First, the perceived value of the data and 
recommendations did not have a statistically significant role in determining if 
recommendations were implemented. However, those programs that had reported the 
most benefit from the AWE generally had higher levels of recommendation 
implementation. Secondly, the data suggest that the ability to tailor activities in the AWE 
contributed to the resulting implementation levels. The data indicates a significant 
relationship between degree of tailoring and the level of implementation, the value of the 
data, and the reported benefits of the AWE. A trend was evident that a program’s phase 
of development also was a factor in receiving benefit from the AWE in that programs in 
or near the mid-developmental phases reported higher levels of recommendation 
implementation and value of data than did programs that were early or late in their 
developmental lifecycle. This trend is illustrated by the dramatic inverted "U" shape seen 
in Figures 18, 20, and 21. This trend is explored further in Chapter V. 

The acquisition manager's definition of the term "benefit" as it applies to AWE 
initiatives is instrumental in conducting further analysis of the outcomes of the AWE. 
The next chapter provides further analysis of the factors contributing to a program's 
ability to benefit from the AWE based on this data as well as collected subjective 


comments. 
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V. ANALYSIS OF CONTRIBUTING FACTORS TO BENEFITING FROM 
ADVANCED WARFIGHTING EXPERIMENTS 


A. INTRODUCTION 


This chapter presents analysis and findings on the contributing factors to 
acquisition managers benefiting from Advanced Warfighting Experiments. The analysis | 
is based on survey ratings data presented in Chapter IV, subjective comments provided by 
the open-ended questions of the survey, and opinions collected during personal 
interviews. The themes discussed in this chapter represent the opinions of a significant 
number of respondents and interviewees. Section B discusses how the level of benefit 
derived from the TF XXI AWE for acquisition managers was limited due to the nature 
and environment of the experiment. Section C defines "benefit" as perceived by the 
majority of acquisition managers and discusses the various benefits received by TF XXI 
AWE participants. Section D discusses the implementation of the Decision to 
Experiment Ladder (DEL) during the AWE and how it effected a program’s ability to 
benefit. The DEL is analyzed in terms of objective documentation, utilization of the 
_ AWE, and a manager’s ability to tailor his initiative’s participation in the AWE. Section 
E provides findings on the effects of program maturity on the ability to benefit. Section F 
presents risk considerations and a tentative risk and benefit analysis plan for acquisition 
managers considering future AWE participation. A chapter summary is included as 
Section G. 


B. LIMITATIONS DUE TO THE NATURE OF THE EXPERIMENT 


According to the experiment’s directive, the Task Force XXI Advanced 
Warfighting Experiment was not designed to provide significant benefits to the 
acquisition manager. The AWE process was designed primarily to determine if 
digitization of the force resulted in increases in lethality, survivability, and tempo [ECC, 
1996]. The nature of the AWE experiment limited the potential derived benefits for the 
program manager. The acknowledged limitations in the design of the AWE experiment 
caused by low repeatability, incomplete data collection, the interaction of effects of 
varying initiatives, and small sample size, precluded any summative findings. Formative 
uses of the AWE by the acquisition team were also limited because, according to one 
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acquisition manager, "large scale exercises do not lend themselves to gathering data that 
is useful." According to many program managers, the experiment’s results were 
influenced by politics. Finally, the applied results of the AWE made many managers 


question the utility of the exercise. 
1. Design of the Experiment 


The design of the AWE, its large scale, and the attempt to combine training with 
assessment, did not allow a controlled, repeatable experiment. Data collection was 
limited to insure that it was unobtrusive to the training unit. The large number of 
interacting initiatives participating in the AWE led to confounded results. Finally, the 
relatively small sample of focused iterations and unrepeatability of the experiment 
prevented the collection of statistically meaningful data. A program manager stated that 
the AWE data and recommendations had low value because they were statistically 
insignificant and thus, invalid. The limitations in the experiment’s design were a result of 


the large scale of the exercise. 
2. Scale of Experiment 


The large scale of the AWE prohibited the detailed evaluation of a system’s 
technical performance. When asked to provide comments that would assist planners in 
designing experiments that would better benefit participating acquisition managers, a 
respondent stated that "large exercises such as TF XXI do not permit the evaluation of a 
system's increase or decrease in combat effectiveness." Another official stated that 
"analysis from an AWE perspective does not usually attempt to discriminate what in 
particular within a program contributed to an overall experimental outcome." The 
previous respondent went on to state that "the amount of equipment required for adequate 
data collection and storage is cost [prohibitive]...In the best cases, 50% of the actions 
performed by the crews is captured. This causes inaccurate conclusions in most cases." 


Another respondent stated: 
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These large scale exercises, that have multiple new systems installed on 
individual vehicles, negate any positive outcome. The soldiers are 
required to learn many new systems and TTPs [tactics, techniques, and 
procedures] at one time and this leads to mass confusion. In many cases 
the systems under evaluation were simply turned off or never turned on. 
These large scale exercises are a political asset and provide no concrete 
data for decision makers to make a decision on the system. 


3. Political Influence 


The influence of politics was a topic discussed by many acquisition managers 
participating in the survey. While "strong political backing" was the lowest rated reason 
that recommendations were implemented, participants believed that the preconceived 
biases of senior army officials effected the final perceptions of a program’s performance 
at the AWE. One participant stated, "watch the politics. On day one of the AWE, the 
event was declared a major success. Contrary opinions were downgraded. This mindset 
started with the then TRADOC Commander. Ethics and testing are a key consideration." 
Another respondent stated that "incoming biases prevented any real analysis of the 
system." The Army’s Applique program, now called Force XXI Battle Command Brigade 
and Below (FBCB2) was often cited as a program that performed marginally in the AWE 
but received the majority of post-AWE funding because "Army leadership backed them" 
and the system was "pushed from the top”. 


4, Utility of Experiment 


Some AWE initiatives received high marks at the experiment but were 
subsequently terminated, leaving the acquisition managers of those programs with 
negative impressions about the utility of the experiment. A program manager for an 
initiative with a large role in the AWE commented that their system had great 
performance but Army leadership did little to exploit the system. A user representative 
for a now terminated program stated: 


The use of [the initiative] in the AWE makes me question the criteria for a 
program’s acceptance into the AWE. If the [initiative] was never going to 
be accepted it should not have been included...Bottom line to us is that 
even though this [initiative] proved very successful, its acceptance was 
doomed from the start. In the end a great deal of time, money, and energy 
was wasted in this effort. 
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For a small scale initiative participating in the experiment, the AWE was 
described as a "lose/lose" situation. The program manager stated that "there were only a 
handful of high visibility programs. While the AWE was a great way of proving out a 
system, the smaller initiatives couldnt get any visibility." In a personal interview, a 
product manager summarized that the AWE was a "diversion." It was "not focused on 
the end goal of getting the product in the hands of the soldiers. The AWE took away 


from program resources to play in a sandbox." 
= Conclusions on Limitations 


Generally, AWEs have not been designed to provide significant benefits to the — 
acquisition managers. Managers charged with the development, production, and fielding 
of programs participating as AWE initiatives must take active measures to increase the 
likelihood of achieving benefits for the program. 


C. BENEFITS TO THE ACQUISITION MANAGER 


The previous section reinforces the fact that the AWE was not designed to 
purposefully provide data to acquisition managers. Nevertheless, survey participants 
identified a number of potential benefits from participation. The extent that a program 
benefited from participation in the AWE varied widely. Chapter IV showed that the 
perceived benefit from the AWE was only moderate, with differing opinions from various 
initiatives. Subjective comments show a wide interpretation of the term "benefit" as it 
applies to the acquisition manager. One respondent stated that "the feedback obtained 
from the AWE was very valuable to the decision making process. It is...easier to 
structure a program based on actual user feedback and very specific user requirements 
obtained during the AWE." From another perspective, a respondent stated that "there was 
no benefit to the program from the AWE. It cost a lot of money to provide system 
support...we feel that the data...will not provide any additional concrete information that 
can be used.” 

A significant number of survey participants agreed that managers must determine 
if potential return from participating in an AWE outweigh the costs. A cost analysis of 
AWE would include expenses for prototyping, manning, fielding, schedule, 
transportation, and other risk categories. One program manager stated: 
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A study should be conducted to define the problems to be resolved and/or 
the goals of the AWE. It should include a section on the perceived 
benefits to each participant. Each PM whose participation is required 
should be given the opportunity to provide an impact statement to his 
program if he participates. He should propose alternatives to minimize the 
impact to his current efforts that have funds obligated. 


The benefits from the AWE to the acquisition manager can be summarized into 
the following categories: (1) marketing and exposure of program, (2) early user feedback, 
(3) refinement of user requirements, (4) development of tactics, techniques, and 
procedures, (5) follow-on support for funding and production decisions, (6) information 
on integration, interfaces, and interoperability, and (7) exposure of developers to the 
user’s environment. 


1, Marketing and Exposure of Program 


Participating acquisition managers were in agreement that no PM would 
participate in the AWE solely for the marketing effects. However, in terms of potential 
AWE benefits, the desire for positive exposure of their program was discussed as a 
benefit by respondents more than any other aspect. 

The importance of being perceived as a "member of the team" was emphasized by 
several respondents. One program manager was adamant that he did not want to 
participate in the AWE but was forced into participation by the user representative. He 
stated that the user viewed the initiative's participation as "a marketing opportunity." 
Another program claimed that they participated in the AWE "to increase visibility of [the 
initiative's] performance capabilities rather than to address developmental issues." A 
program that reported a very high level of benefit from the AWE stated that benefit was 
"extremely high from a marketing perspective. The [initiative] demonstrated to the world 
its ability to dominate the battlefield." 

Marketing was most often described as an ancillary benefit. In a personal 
interview, a senior engineer stated that the AWE process is a good way to expose the 
program but was not just marketing. Participation provided exposure of the program 
among soldiers and Army leadership. Another participant said in an interview, that the 
positive response to his program benefited him. He added, however, given the nature of 
the experiment, the results and data could not be valid. 
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2. Early User Feedback 


Most acquisition programs do not experience any hands-on exposure to actual 
users until Initial Operational Test and Evaluation (OTE). A program manager stated 
that any time you can get your product into the hands of the actual user early, it serves to 
benefit the program. IOTE was also characterized by a significant number of respondents 
as "too structured” to provide any significant feedback from the user. Feedback during 
IOTE is also "too late to impact system design." 

It is still unclear if benefits associated with the transference of AWE data to 
reduce the operational test and evaluation process will be realized. In February 1997, an 
article discussing this subject stated: | 


Whether or not the results from TF XXI will contribute to the formal test 
and evaluation...[process]...is a question under debate right now...The 
acquisition community has averred that the experiment data will not 
receive formal consideration, but at the same time we’re hearing 
comments that the testing community needs to be cognizant of the 
participation in TF XXI and take advantage of the test results coming out 
of that. [Lum, 1997, p26] 


3. Refinement of User Requirements 


For the majority of respondents, participation in the AWE provided significant 
data to be used for the refinement of user requirements. Feedback provided from 
representative users experimenting with the new systems in a combat simulated 
environment was determined to be of high value to many acquisition managers. 

For one program, the AWE caused a complete reanalysis of requirements and led 
to the development of new key performance parameters. One manager stated that "the 
AWE provides a good forum to ascertain whether or not certain requirements are feasible 


or even desirable from a user perspective." A user representative summarized: 


The AWE demonstrated a great deal of utility in the requirements 
development process. The Army's experimental process determined what 
concepts and technologies worked and how they can be made to work 
better. The process also identified concepts that did not work well. 


A program manager directing a pre-milestone II initiative stated that the AWE 
provided tremendous insight for program definition and risk reduction. All of the 


associated AWE recommendations for this initiative were fully implemented. A user 
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representative of a program highly concerned with refinement of requirements stated that 
"the lessons learned from actual field user are invaluable." 

Refinement of requirements was even beneficial to some programs already in 
production. One particular initiative utilizing extensive commercial-of-the shelf and non- 
developmental technology had received a Milestone III decision and moved on to 
production without any operational testing. Participation in the AWE resulted in "major 
changes in system architecture". According to the program manager, the experiences 
gained would be "utilized in the design of follow-on systems." 

Regardless of an initiative’s level of maturity, most acquisition managers were 
able to collect valued data that would assist in the refinement of system requirements, and 
contribute to the development of a more effective system. 


4. Development of Tactics, Techniques, and Procedures (TTPs) 


The development of tactics, techniques, and procedures (TTP) was identified by a 
Significant number of respondents as a benefit received from the AWE. TTP 
development is not normally a function of the acquisition manager, but rather a function 
of the branch specific office responsible for organizational doctrine. However, the 
development of TTP is a vital step in the process of gaining user support for a technology. 
The program manager for one AWE system described his initiative as a whole new 
concept for the Army. The AWE helped to refine tactics, techniques, and procedures in 
an area that previously had no doctrine. Another manager stated that the AWE confirmed 
"a concept in an operational environment without a commitment to expensive, pre- 
production hardware." 


— Follow-on Support for Funding and Production Decisions 


The nature of today’s defense acquisition environment includes a constant struggle 
for program funding. A former acquisition manager stated that he spent twenty-five 
percent of his time as a PM dealing with funding issues on his program. Many 
respondents stated that one of their goals in AWE participation was to gain data to 
support funding requests and to "get a foot in the door for production funding". 

The Army’s Warfighting Rapid Acquisition Program (WRAP) was tied to the TF 
XXI AWE. The WRAP process was designed to accelerate the fielding of advanced 
systems and technologies from AWEs, Advanced Technology Demonstrations (ATDs) 


65 





and Advanced Concept Technology Demonstrations (ACTDs). WRAP links TRADOC’s 
battle labs to the formal acquisition process so that compelling successes quickly 
transition from experimentation to enhanced warfighting capability. The TRADOC 
Commander nominates candidates for WRAP after a review based on urgency of need 
and compelling success. Approved candidates are then ranked by priority and given two- 
year funding for operational prototypes. [Monnett and Rheinlander, 1996] 

Several AWE initiatives desired publicity to secure WRAP funding and used | 
AWE results as a demonstration of compelling success. A program that was ultimately 
selected for WRAP commented that "the data and recommendations gained from the 
AWE were critical to the system’s success as a WRAP candidate. Without the WRAP 
funding, the program was an unfunded requirement with little probability of being 
funded." | ; 

The attainment of production funding was of particular concern for several 
initiatives with no programmed dollars beyond the AWE. The Department of the Army 
specifically delayed production decisions on some programs until the AWE was 
completed. During a personal interview, a PM stated that Army leadership “expected 
great things of the AWE." The manager of a now terminated program commented "if the 
program could have shown merit at the AWE as a last chance effort, then it would have 
- been a good investment. We had nothing to lose." 

The data show that initiatives facing future funding decisions stand to receive 
significant benefit from AWEs if compelling success can be demonstrated in an AWE. 


6. Information on Integration, Interfaces, and Interoperability 


Integrating, interfacing,: and interoperability of new technologies within the 
current Army force structure is a constant challenge among acquisition programs. AWE 
participants collected data to meet future integration, interface, and interoperability 
requirements. This data provided significant benefit to acquisition managers. 

Integration in acquisition is defined as the incorporation of a system as an equal 
into an organization. A system interface is a common boundary between two associated 
systems across which information may flow. Interoperability is defined as the ability of a 
system to provide services to and accept services from other systems to enable them to 
operate effectively together. [CALL, 1999]. 
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One program manager explained how AWE participation benefited the program in 
terms of integration, interfaces, and interoperability: 


Historically each PM had his own requirements to implement and did not 

feel justified in experimenting with interoperability. Each PM would 

develop an interface to other systems using the minimum resources to get 

the job done. The interface was implemented from a paper design based 

on the data that needed to be shared. In the AWE each [initiative] got the 

opportunity to view the operational picture as presented by each of the 

other [initiatives]. 

According to program managers, operational testing does not provide the required 
data to facilitate integration, interfacing, and interoperability. The AWE, however, 
provides much needed information on how a program “interfaces with the force." 
Reportedly, the AWE was a "great way of proving out hardware" and a "great way to test 
integration." A program manager stated that it is "always worthwhile to exercise 
interoperability with other systems to identify nuances and characteristics of other 


systems that must be ‘designed to’." 
7. Exposure of Developers to the User’s Environment 


By participating in the AWE, acquisition managers and program engineers 
received exposure to the user’s environment in a realistic combat simulation. Most of the 
participating managers lacked any real experience in military field operations. This 
exposure was cited as a significant benefit from AWE participation. 

During the AWE, program managers were allowed to observe the experiment 
firsthand. A participating program manager stated that the AWE was beneficial because 
it "allowed for realtime feedback to answer questions and understand deployment issues." 
A senior official with oversight of multiple initiatives in the AWE stated during an 
interview that the AWE had "additional intangible benefits" to the program manager, 
beyond the benefits derived from official collected data. The AWE provided acquisition 
managers "early exposure to the customer." According to the official, most of his 
engineers and program managers have never been on active duty. The AWE provided an 
opportunity for these personnel to experience the user's environment firsthand. The AWE 
also allowed the engineers and program managers to gather data on peripheral programs 
and to learn the state of technology across the force. Another program manager stated 
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that "the biggest benefit was from debriefing the soldiers that used the equipment. They 
provided more information, all be it subjective, than any questionnaire did." 

The data clearly show that acquisition managers who lack significant experience 
in military field operations are positioned to receive significant benefit from the AWE by 
gaining an understanding of the environment and scenarios in which their respective 


systems will operate. 
D. THE DECISION TO EXPERIMENT LADDER (DEL) 


The decision-to-experiments ladder (DEL) discussed in Chapter II, directly links 
experiments to decisions and is designed to increase the utility of experiments. The DEL 
includes: issues, decisions to be made, argument to support the decisions, hypotheses to 
be adjudicated, experiments to resolve hypotheses, and analysis and measures to 
implement decisions. [Lucas, Banks, Vye, 1998] — 

The level that the DEL was implemented in the AWE appeared to directly effect 
an initiative’s ability to benefit from the experiment. The analysis of survey responses 
provides evidence that those program managers who could provide a detailed explanation 
of the specific developmental issues being addressed on their program at the time of its 
participation in the AWE; the decisions to be made from gathered data, and were able to 
specifically tailor their program’s activities and measures in the AWE to support these 
issues and decisions, generally reported a much higher level of perceived benefit from the 
experiment. 

The extent to which acquisition managers implemented the DEL in the TF XXI 
AWE can be measured by analyzing the extent a manager was able to tailor or influence a 
program’s specific activities in the AWE to reflect the program’s current acquisition issues 
and decisions. Additionally, one can analyze the level at which AWE data was derived 
and used to support future acquisition decisions. 


1. The Ability to Tailor 


Chapter IV illustrated that the extent a manager was able to tailor or influence a 
program’s specific activities in the AWE to reflect the program’s current acquisition issues 
and decisions had a direct impact on the extent to which the program benefited from 
participating. Those initiatives reporting a high level of ability to tailor or influence 
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generally reported much higher levels of perceived benefit from participation in the 
AWE. 

The level that acquisition managers were able to participate in the planning 
process for the TF XXI AWE, as run by the TRADOC Joint Venture office, varied among 
programs. It is unclear if the level of participation in the planning process for specific 
programs was dictated by the various empowered organizations including TRADOC, 
TRADOC Analysis Center White Sands (TRAC-WSMR), the Operational Evaluation 
Command (OEC), and the Test and Experimentation Command (TEXCOM) or if the 
level of participation was a conscious decision made by the program managers and user 
representatives for the participating initiatives. 

Many of the surveyed Initiatives reported that they had no influence on their 
system’s role in the experiment or on the analysis plan for their system. One program 
manager stated that "there was a hands off policy instituted" and that "there was no 
avenue at all for any program influence [on the AWE plan]." Other managers reported 
that they had no real influence on the conduct of the AWE because they "had no issues to 
be addressed." Another manager reported that "the requirements from the [user 
representative] did not line up with the main thrust of the AWE," therefore no tailoring of 
the AWE was attempted. Funding shortages also contributed to the limited tailoring 
reported by mangers. No additional funds were provided to any program specifically to 
support participation in the AWE. This required managers to "minimize the AWE’s 
impact on current efforts that already had funds obligated." 

The inability of one program manager to influence his program’s activities in the 
AWE resulted in the initiative not using "real communication systems." The initiative 
was not "employed over realistic distances." As a result, "times and completion rates 
were unrealistically skewed." The initiative’s level of benefit from the AWE was 
subsequently reported as low. A user representative for a different program stated that he 
had "little influence on how his system was used in the AWE" and this "hampered the 
[initiative’s] ability to fully participate." Another user explained that he was unable to 
influence his program’s role in the AWE and that his initiative was "artificially restricted.” 
The result was a failure to follow establish tactics, techniques, and procedures leading to 
a low reported value for the gathered data. | 

Some programs, however, reported a high ability to tailor or influence their 
initiative’s participation in the AWE. One user representative stated that they were "able 
to tailor all of the required activities" because they were "part of the AWE IPTs 
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[Integrated Product Teams]." They reported that they played an active role in developing 
the analysis plan for OPTEC to conduct their assessment. They also assisted in the data 
collection and assessment process. Another user representative stated that his office was 
an active participant "in the various working groups by the primary contractor, PM, and — 
the TSM [TRADOC System Manager] to ensure that all exercise objectives were met or 
exceeded." | 

Generally, those initiatives reporting a high level of ability to tailor or influence 
reported much higher levels of perceived benefit from participation in the AWE. One 
program manager summarized that "we pretty much had our way. They reported their 
perceived benefit from the AWE as very high. The initiative that participated in the 
AWE IPTs and contributed to the analysis plan discussed earlier, was reportedly praised 
by the then TRADOC commander when he commented that the system was a "clear 
winner" in the AWE. | | 

Both program managers and user representatives agree that to achieve maximum 
benefit from AWEs, acquisition managers must be able to participate in the planning 
process. One manager stated that acquisition officials should participate in the planning 
process to "define clear goals and objectives, as well as the process to be used to achieve 
the goals and objectives.” Another PM stated that "a study should be conducted to define 
the problems to be resolved" and that it should “include a section on the perceived 
benefits to each participant." 

The TRADOC Joint Venture office has recognized the value of having program 
managers participate by extending their role in the planning process for the Joint 
Contingency Force (JCF) AWE to be held in late 1999. A program manager participating 
in this upcoming AWE stated that he developed Measures of Effectiveness (MOEs) for 
the AWE in concert with independent analysis agencies. For the JCF AWE, OPTEC 
representatives will meet with program managers to determine issues that are linked to 
AWE objectives [TRADOC JCF-AWE, 1998]. Still, no additional funding is provided to 
the program manager for AWE participation or for implementation of any derived 
recommendations. Participation in AWEs requires expensive prototyping, manning, 
fielding, training, and transportation costs that must be drawn from existing research and 
development accounts. Program managers cannot increase their roles in the AWE 
process and ability to tailor activities without dedicated support from the Army's 
budgetary process. Therefore, they may likely continue to report low levels of benefit 


from live experiments. 


70 











2. Documentation And Use Of AWE Data 


The level at which AWE results were documented within the acquisition 
management office had a direct effect on the ability of a program to benefit from 
participation. As discussed in Chapter II of this thesis, the value of any evaluation must 
be judged by its eventual utility in implementing change. 

A significant number of the TF XXI AWE initiatives had no historic data or 
documentation of their AWE experience. Several programs were not able to confirm that 
their system participated in the TF XXI AWE. Six of the original seventy-two offices 
initially identified to participate in the AWE admitted that they had no records or 
knowledgeable managers that could comment on the experiment. Personnel turnover and 
attrition can account for some of this absence of knowledge. When asked to list specific 
issues that needed to be investigated in the AWE, a user representative stated 
"unfortunately, we are not aware of what specific developmental issues were being 
addressed on the program. The personnel involved in them at the time have PCS’d 
[moved due to a permanent change of station] and that data is not available." A 
substantial majority of the programs, however, had significant records and detailed after- 
action reports from the AWE. | 

A respondent discussed the AWE documentation process and the eventual utility 
of the data. He reported that "too often a unit pushes to get a system in an AWE and 
never comes into the decision making process nor does guidance get passed down to the 
people who have to execute the actions to make it happen." Another program office 
admitted their shortcomings by stating "we needed to figure out a better way to document 
[lessons learned from the AWE]." Another manager recommended that participants 
"document as well as possible the lessons learned so that a third party can understand." 

_In some cases, implementation of the DEL in the TF XXI AWE was limited due 
to poor documentation before and after the experiment. In many cases, issues and 
decisions to be made were not identified prior to the AWE. One program manager stated 
that his system "did not have any specific developmental issues that needed to be 
addressed in the AWE." Another program manager stated that "no specific data was 
proposed to be gathered or evaluated, nor were any specific issues identified for analysis." 
This lack of issue identification prevented acquisition managers from developing 
arguments and hypotheses to be adjudicated. Many programs did not perform analysis or 
develop measures to implement decisions. One program manager stated repeatedly that 
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"the recommendations fell outside the responsibility of the PMO [Program Management 
Office].". Another program manager stated "to date, we have not seen the [AWE] report, 
and have only heard rumors as to the outcome." The manager continued that "there was 
no benefit to the program from the AWE." 

A manager from an initiative reporting a high level of perceived benefit 
recommended that AWE participants "define clear goals and objectives, as well as the 
process to be used to achieve the goals and objectives." Another program reporting very 
high benefit, contributed the benefit to "thorough planning and understanding of what the 
expected outcomes were supposed to be." A program with a low level of reported benefit 
explained that their problems related to a failure to identify how collected data would "be 
used to focus future activity." | 

One program manager effectively summarized the requirement for documentation 
and planning by stating "it is important for a program manager to develop test objectives 
and measurement processes that identify product improvements, areas for future research, 
and follow-on programming, while allowing AWE results to drive go/no-go decisions for 
the Army." 


3. Conclusions on the Decision to Experiment Ladder 


The data clearly indicates that the benefit derived from participation in AWEs can 
be increased by managers developing detailed experiment objectives and expected 
outcomes, systems for data documentation and analysis, and strategies for implementation 
of AWE data and recommendations. Additionally, a program’s specific AWE activities 
should be tailored to reflect these objectives, systems, and strategies. 


E. PROGRAM MATURITY 


As illustrated in Chapter IV, a relationship exists between an initiative's 
programmatic maturity and the benefit gained from participating in the AWE. More 
maturity does not directly transcribe into a higher ability to benefit, but rather a point 
exists where a program is best positioned to benefit from experimentation. All of the 
collected objective data supports the finding that an initiative in the mid-developmental 
phases of acquisition will be best positioned to benefit from AWE participation. 

Acquisition managers agree that timing is critical to success in an AWE. 
Programs in Concept Exploration (CE) are too immature to effectively participate. 
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Programs in production cannot take advantage most findings or recommendations from 
the AWE. Even OPTEC agrees that "AWE planners should carefully consider whether it 
is worthwhile to devote scarce data collection and assessment resources to assess post- 
Milestone II systems" [ECC, 1997]. 

A key factor of maturity, as defined by survey participants, was system reliability. 
Immature initiatives were generally not environmentally hardened or ruggedized for field 
trials. Poor operator handling of these non-ruggedized systems resulted in excessive 
maintenance problems that were reported as system failures and contributed to a negative 
post-AWE analysis. A program manager commented that the AWE was "not a good 
environment for marginal systems." According to this manager, if the system is not 
sufficiently mature, "the opportunity for failure is much greater than the opportunity for 
success." 

Participating initiatives must also be sufficiently mature, programmatically, so 
that prototypes and demonstration versions are readily available for participation. In 
several cases, surrogate systems were included in the AWE to simulate proposed 
capabilities. Often these surrogates did not effectively represent proposed production 
systems. A program manager stated that immature systems were responsible for 
"reducing the effectiveness or otherwise limiting the lessons learned by other platforms." 
Respondent comments show that the varying levels of maturity and functionality of 
initiatives affected the overall performance of the experimental force." It was stated that 
“soldiers are practical and will not waste time using technology that is cumbersome or 
does not work As a result, substandard systems were ignored." 

Some mature programs that had moved into production prior to the AWE were 
able to benefit from refinement of Tactics, Techniques, and Procedures (TTP) at a higher 
level then immature programs. Production items generally provided the reliability 
required to adequately stress the system using realistic methods. Program managers 
agreed that systems providing "new capabilities" or "leap-ahead Pe anogiers were best 
situated to benefit from the development of TTPs. 

Program managers often lacked the funding required to implement changes 
recommended by the AWE analysis. One program manager stated that "programs must 
be sufficiently mature to preclude the need for drastic changes as a result of the AWE." 
As stated earlier, one initiative was forced to make "major changes in system 
architecture" as a result of the AWE. Funding for this restructuring was made available at 
the expense of other programs within the program management office. 
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Survey comments show that programs at any level of maturity can benefit from 
AWE participation. Programs that are early in development, such as CE and PDRR, can 
demonstrate the benefits of new concept technologies to the force, and gain subsequent 
program support. Mature programs that are in production, can gain visibility and positive 
marketing aspects required for continued funding. However, initiatives in mid- 
developmental phases of acquisition, that are sufficiently mature to tolerate the harsh 
environment of a live experiment but do not have a finalized architecture, seem .to benefit 
most from an AWE. 


F. RISK AND BENEFIT ANALYSIS 


Survey comments indicate a substantial risk to programs participating in an AWE. 
Program risks include factors beyond the costs of participation, to include a poor return 
on investment, potential negative exposure, and extensive changes in requirements. 
Acquisition managers contemplating participation in future AWEs could benefit from 
conducting a detailed analysis of the risks associated with AWEs, comparing these risks 
to potential benefits, and considering their tolerance for risk as a factor in making a 
decision to participate. 

Program managers reported a wide range of tolerance for risk. One program 
manager stated "we had nothing to lose." The program had no funding for future 
production and the program manager believed that the AWE served as a "last chance" 
effort. Reportedly, if the initiative could have shown merit at the AWE then participation 
in the experiment would have been a good investment. Another manager stated "we had 
nothing to gain." The design of the system was finalized and a production decision had 
been made and funded. 

Data collected from TF XXI AWE participants can be used to measure the risks 
and benefits involved with AWE participation. This thesis presents a model to measure 
risk as compared to potential benefit. Although this model has not been validated 
through actual implementation, it is supported by the findings of this thesis. Program 
managers should complete this risk and benefit analysis, and if necessary, modify the risk 
analysis plan based on the unique situation of their program and their personal tolerance 
for risk. Potential risks involved with AWE participation are presented, followed by a 
description of risk and benefit factors. A method to compare risk factors and potential 
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benefits and analyze the risk situation for AWE initiatives is illustrated. Finally, methods 
to mitigate some of this risk are presented. 


1. Potential Risks 


Potential risks are areas where acquisition programs may be negatively effected. 
This thesis has reveled three significant risks involved with participation: poor return on 
investment, potential negative exposure, and extensive change in requirements. 


a. Poor Return on Investment 


TF XXI AWE acquisition managers stressed that AWE participation is 
resource intensive. One manager stated that "PMs must determine if potential return 
outweighs the cost in prototyping, manning, fielding, transportation, etc." He stated that 
the Army should provide funding to program offices for participation. However, no 
funding has been provided for past AWEs and as stated in the plan for the upcoming Joint 
Contingency Force (JCF) AWE, submitting agencies must "bear the complete costs 
associated with the initiative" [TRADOC JCF-AWE, 1998, p3]. Given the low levels of 
benefit reported by some programs, there is an inherent risk that the benefits received by 
any one initiative will not justify the cost. 


b. Potential Negative Exposure 


Program exposure to both user and senior Army leadership and the 
associated marketing effects of AWE participation must be considered with both their 
positive and negative potential consequences. Some initiatives with outstanding AWE 
performance received high acclaim and significant support for future funding decisions. 
However, this thesis has many examples of programs that received negative feedback 
from the AWE. The results of the AWE provided negative marketing for some programs 
that hampered their ability to move into production. One initiative had two of four 
systems fail at the AWE. This was reported as a reliability problem when damage was 
likely due to poor operator handling. Another system received a poor evaluation 
primarily due to host vehicle failures. The risk of program exposure, positive or negative, 
must be considered prior to AWE participation. 
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C. Excessive Changes in Requirements 


One of the stated goals of the TF XXI AWE was to provided information 
to support investment decisions and refinement of requirements [ECC, 1996]. Some TF 
XXI initiatives received so much information that the refined requirements negated the 
current system design. These programs were then either terminated, or underwent 
complete restructuring. One such program was in production at the time of the AWE and 
subsequently was forced to design a complete new architecture and develop new metrics 
to measure performance. Although these changes may represent an ultimate benefit from 
AWE participation, some acquisition programs may not be positioned to adequately adapt 


to such drastic changes in requirements. 


2. Risk Factor Ratings 


Risk factors are those issues that can be measured by an acquisition manager in an 
attempt to determine overall risk from participation. As shown in Table 2, these risk 
factors directly contribute to the three identified risks of poor return on investment, 
negative exposure, and excessive changes in requirements. This thesis has revealed seven 
contributing risk factors to AWE participation including: (1) system maturity, (2) 
ruggedization and maintainability, (3) how the system fits within the focus of AWE, (4) 
funding availability, (5) equipment availability, (6) incoming biases, and (7) the status of 


production decisions. 


Table 2. Factors Contributing to Potential Risks 
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Maturity is measured in terms of the system’s programmatic stage of 
development, from Concept Exploration to Production. Maturity effects the ability of a 
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program to perform adequately as well as the ability of the initiative to implement 
recommendations derived from the AWE. Programs in Concept Exploration are high risk 
because of their unpredictability and vaguely defined roles and requirements. Programs 
in Production or Limited Production are considered medium risk because they may be too 
far in development to capitalize on recommendations. Programs in PDRR or EMD are 
considered low risk. 


b. Ruggedization and Maintainability 


Ruggedization and maintainability are measured as high or low. A system 
that is highly rugged is able to withstand the stresses associated with operational use in 
harsh environments. A system that scores low in this factor has sensitive components 
with maintenance procedures that are difficult to conduct in a field environment. Low 
ruggedization may be associated with early prototype systems. 


C. Focus of the AWE 


A manager must analyze the published goals and objectives of the AWE as 
Stipulated by the planning officials and characterize how his program fits within the 
experiment focus. Also, a manager must measure how the AWE will address the issues 
facing his initiative. How an initiative fits into the focus of the AWE is measured as 
complimentary or ancillary. As demonstrated in the TF XXI AWE, those programs that 
are not a high priority of the analyzing agency may not receive adequate focus and 
feedback. This inadequate focus and feedback can lead to a poor return on investment. 


d. Funding Availability 


Participation in an AWE is resource intensive. A program must have 
sufficient funds for prototypes, manning, training, transportation, and maintenance. 
Required funding will vary based on the complexity of the system. Additional funds will 
not normally be available specifically for AWE participation. Low availability of funding 
can contribute to an inability to properly prepare for the AWE and an inability to 
implement any derived findings. This can result in negative exposure and low return on 
investment. Funding is measured as sufficient or insufficient. Sufficiently funded 
programs will have budgeted dollars specifically for AWE participation and are 
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considered low risk. Insufficiently funded programs will have tight budgetary constraints 
and are considered high risk. 


é. Equipment Availability 


Development of prototypes or surrogates for AWE participation requires 
additional costs, time, and developmental testing beyond that normally experienced in 
system development. Surrogates and prototypes may also be less robust and contribute to 
poor performance at AWEs, potential negative exposure, and low return on investment. 
Mature systems with readily available production items are best positioned for AWE 
participation. Equipment availability is measured on a continuum from production items 
to surrogate representatives. Initiatives with readily available production models are 
considered low risk. Initiatives that require the development of prototypes or surrogates 


are considered high risk. 
f. Incoming Biases 


Program managers from the TF XXI AWE stated that incoming biases by 
users, evaluators, and Army leadership can prevent any real analysis of a system’s 
performance at the AWE. Biases are measured as either favorable or unfavorable. 
Unfavorable biases can contribute to reported poor performance and negative exposure. 
Those programs with unfavorable biases are considered high risk. Those programs with 


favorable biases are considered low risk. 
g. Status of Production Decisions 


Production decision status is measured as either pre-production with 
Milestone II decision, imminent production decision, or post production. Programs that 
have secured approval and funding for production prior to participation in an AWE but 
have not yet begun production, face the added risk of poor performance and loss of 
support. These pre-production decision initiatives are categorized as high risk. Those 
initiatives with imminent production decisions are considered medium risk, in that AWE 
performance can significantly influence the survivability of the system. Programs in post- 
production or with no pending production decision are categorized as low risk. 
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3. Benefit Factor Ratings 


Benefit factors are those issues that can be measured by an acquisition manager in 
an attempt to determine the potential to benefit from AWE participation. These factors 
can be measured in terms of potential to benefit based on the unique needs and situation 
of the initiative in question. This thesis has revealed seven benefit factors to AWE 
participation including: (1) marketing and exposure of program, (2) early user feedback, 
(3) refinement of user requirements, (4) development of tactics, techniques, and 
procedures (TTP), (5) follow-on support for funding and production decisions, (6) 
information on integration, interfaces, and interoperability, (7) exposure of developers to 


the user’s environment. 
a. Marketing and Exposure of Program 


The effects of marketing and exposure of a program are a result of system 
performance at the AWE and cannot be accurately predicted. The potential for benefit 
from advantageous exposure is measure based on needs of the program. A program that 
requires positive exposure for survival, would rate this area as high potential benefit. 
Those programs that do not require this marketing effect, or already have strong support 
among senior leadership, would rate this area as low potential. 


b. Early User Feedback 


The potential to benefit from early user feedback is measured based on 
needs of the program. Those programs that have had little to no user feedback would rate 
this area as high. Those programs that already possess sufficient feedback, or those that 
are not in a position to implement any recommendations from the feedback, would rate 
this area as low potential. 


c. Refinement of User Requirements 


The potential to benefit from refinement of requirements is measured 
based on the initiative's stage of development. Those programs early in Concept 
Exploration would score this area as high, because they are best positioned to implement 
the refinements without a significant investment. EMD programs would score medium. 
Programs late in development or into production would score this area as having low 


79 


potential benefit because the refinements cannot be adequately implemented in 


production models. 
d. Development of Tactics, Techniques, and Procedures (TTP) 


The potential to benefit from TTP development is measured based on 
needs of the program. Initiatives that are new concept technologies with no current TIP 
would rate this area as high. While existing TTPs can be modified and refined based on — 
AWE data, programs that are not new concepts to the Army and have established TTPs 
do not require this data to move into production. These initiatives would rate this area as 


low potential benefit. 
é. Follow-on Support for Funding and Production Decisions 


The need for funding and support for production decisions determines the 
potential benefit in this area. Those programs with production decisions pending would 
rate this potential benefit as high, as favorable AWE performance can result in favorable 
decisions by Army leadership. Those programs with future production decisions would 
rate this area as medium, as the derived AWE data can be used to support these decisions. 
_ Programs that are currently in production have no extensive need for funding support and 
would rate this as a low potential benefit. 


f. Information on Integration, Interfaces, and Interoperability 


Potential benefit in this area is measured based on need. Those systems 
with a high requirement to integrate or interface with other programs would rate this area 
as high. An initiative that does not interface or interoperate with other systems would 


score this as a low potential benefit. 
g. Exposure of Developers to the User’s Environment 


The exposure of developers to the user’s environment is rated as a high 
potential benefit for those programs where managers and engineers have little field 
experience. Those programs with managers and engineers who have sufficient exposure 
to the user’s environment would rate this potential benefit as low. | 
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4. Measurement and Analysis of Risk and Benefit 


a. Risk Measurement 


The presented risk factors can be subjectively measured and totaled using 
the risk analysis matrix shown as Table 3. In Table 3, an acquisition manager should 
subjectively measure each of the seven risk factors as either high, medium, or low, and 
circle the appropriate rating. The total number of ratings for each of the three 
measurements should be totaled and entered as the factor total. Each of the three factor 
totals should then be multiplied by a risk multiplier with the high risk factor total 
multiplied by three, the medium risk factor total multiplied by two, and the low risk factor 
total multiplied by one. The results of this multiplication should then be entered as the 
risk total for each of the three measurement levels. All three risk totals should be added 
together to compute the total risk score. 

| The results of a hypothetical AWE initiative is presented in Table 4 as a 
risk factor example. This hypothetical initiative is a new digital communications system. 
This initiative is currently in the PDRR phase of development, so maturity is assessed as 
low risk. The system has not been extensively ruggedized but has low maintenance 
requirements, therefore, ruggedization / maintainability is assessed as moderate risk. The 
focus of the AWE is digital communication systems so the initiative is considered 
complementary and risk is assessed as low. Funding is available for research and 
development on this item, though specific funds have not been programmed for the 
AWE. Therefore, the funding availability risk factor is assessed as moderate. A 
prototype of the system is being developed specifically for the AWE, so the equipment 
availability risk factor is rated as high. The Army leadership believes that this 
communication system is of paramount importance to the future of army digitization and 
has a high level of favorable biases. Therefore, the bias risk factor is assessed as low. 
The program believes it will move into EMD shortly and be facing a production decision 
within the next two years, and thus has assessed the production decision risk factor as 
medium. The risk factor totals equate to one high risk factor, three medium risk factors, 
and three low risk factors. The risk multipliers are applied for a total risk score of 12. 


81 
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Table 4. Risk Factor Example 


Not Rugged / High 
Maintenance 
Ancilla 
Tight Budget 
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Production Items 


— Available 





b. Benefit Measurement 


Potential to derive benefit from the AWE can be measured based on the 
factors presented earlier in this chapter and totaled using Table 5. In Table 5, an 
acquisition manager should subjectively measure each of the seven potential benefit 
factors as either high, medium, or low, and circle the appropriate rating. The total 
number of ratings for each of the three measurements should be totaled and entered as the 
factor total. Each of the three factor totals should then be multiplied by a benefit 
multiplier with the high benefit factor total multiplied by three, the medium benefit factor 
total multiplied by two, and the low benefit factor total multiplied by one. The results of 
this multiplication should then be entered as the benefit total for each of the three 
measurement levels. All three benefit totals should be added together to compute the 
total benefit score. 
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The results of a benefit factor analysis of a hypothetical AWE initiative is 
presented as Table 6. This example uses the same hypothetical initiative presented above, 
in risk factor computation. The program manager has assessed his potential to benefit in 
each of the seven benefit factors. The program manager has determined that his program 
has a moderate need for marketing and exposure. This program does not have a wide 
familiarity among Army leadership, yet this exposure is not required for program survival 
in its current phase. Therefore, he has assessed the potential for a marketing and 
exposure benefit as medium. The program manager has determined that his initiative 
requires early user feedback from its user community and has assessed the potential to 
benefit from in this area as high. The initiative is currently in PDRR and has a medium 
potential to benefit from requirement refinement while at the AWE. While some TTPs 
exist for digital communications system, the program manager has determined that they 
have a moderate need to refine these TTPs and has assessed the potential to benefit in this 
area aS medium. The program manager believes the initiative will move into EMD 
Shortly and be facing a production decision within the next two years, and thus has 
assessed the potential to benefit from funding support and future production decisions as 
medium. Being a digital communications system, the program requires detailed 
information on integration, interfaces, and interoperability, and thus has assessed this 
potential benefit as high. The program managers and engineers currently have little 
exposure to the user’s field environment, so this factor has also been assessed as having a 
high potential benefit. The results of the benefit assessment included three areas of 
potentially high benefit, four areas of medium benefit, and one area with low potential 
benefit to the program. With the benefit multipliers applied, the total benefit score 
computed to seventeen. 


Cc. Comparison and Analysis 


Resulting risk and potential benefit measurements can be compared using 
the risk analysis chart illustrated below as Figure 22. When computed risk and benefit 
scores are compared, four risk analysis quadrants are developed. Quadrant I describes an 
initiative with a high potential to benefit from AWE participation with very low risk to 
the program. Program managers assessing their system in Quadrant I are in the best 
position to benefit from AWEs. Quadrant DJ describes a program with very low potential 
benefit and high associated risk. From an acquisition manager’s perspective, these 
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programs would be best served by not participating in AWEs. Quadrant II and Quadrant 
IV describe programs with situations that are not as clear as the previous two cases. A 
program in Quadrant II has a high potential for benefit but not without significant risk. In 
this quadrant, an acquisition manager must determine his tolerance for risk in making an 
AWE participation decision. A program in Quadrant IV has low risk from participation 
but also does not stand to receive significant benefit. Program managers in this situation 


should base their participation decision on a monetary basis or consider other potential 


benefits not discussed in this thesis. 


Table 6. Benefit Factor Example 
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The results of a risk and benefit comparison and analysis for a hypothetical 
AWE initiative are shown in Figure 23. The hypothetical initiative is the same 
communication system used above for the risk and benefit factor measurement. The 
system's total risk score was twelve and the total benefit score was seventeen. As shown 
in Figure 23, the resulting comparison lies in Quadrant II of the risk / benefit comparison 
chart. This signifies that the initiative has high potential benefit from AWE, but with 
moderately high risk. Based on this analysis, the initiative’s acquisition manager must 
determine his tolerance for risk in making an AWE participation decision. Through the 
application of risk mitigation, as discussed below, the risk score can be reduced so that 


the initiative lies within Quadrant J, where potential benefits are high, but participation 








risk 1s low. 
21 
Quad I Quad I 
High Potential High Potential 
Benefit / Low Benefit / High 
Risk Risk 
Potential 10.5 
Benefit Score 
Quad IV Quad Il 
Low Potential Low Potential 
Benefit / Low Benefit / High 
Risk Risk 
0 
10.5 21 
Risk Score 


Figure 22. Risk / Benefit Comparison and Analysis 
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Risk Score 
Figure 23. Risk / Benefit Comparison and Analysis Example 


5. Risk Mitigation 


The risks discussed previously in this chapter are based on reported results of the 
TF XXI AWE. Programs participating in future AWEs can mitigate these risks based on 
the TF XXI experience. Methods to mitigate risk include program budgeting, marketing 
to the user, user training, and participation in the planning process. 


a. Program Budgeting 


The experiment plan for the JCF-AWE states that one the main criteria 
used for consideration for inclusion is "the ability of the submitting agency to bear the 
complete costs associated with the initiative, to include its integration into the JCF-AWE 
architecture", [TRADOC JCF-AWE, 1998, p3]. A manager participating in the TF XxX] 
AWE stated that he needed to conduct "monetary trade-offs between supporting his 
current mission and supporting the AWE." Certainly, program managers do not need to 
be the sole source for AWE funding. Stakeholders in the system, to include program 
managers, combat developers, TRADOC system managers, and battle labs can all 
combine resources to fund AWE participation. AWE participation risks centered on 
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funding issues can be mitigated by proactively budgeting for AWE participation early in 
the system’s lifecycle using available research and development dollars. This proactive 
budgeting will help ensure that sufficient funds are available for AWE participation 
without the need to conduct schedule or performance trade-offs. Certainly, as the likely 
benefit of AWE participation can be enhanced, program managers will be more motivated 
to build participation in their program planning. 


b. Marketing to User 


In several cases, initiative managers stated that their system received a 
poor assessment at the AWE because actual users failed to appreciate the capabilities of 
the system. One such initiative reported that it did not get much use at the AWE because 
the soldiers were concerned with some potential negative characteristics of its design. He 
stated that the soldiers using the equipment did not fully understand these characteristics 
and their effects. Marketing the advantages of the system to the soldier to enhance the 
perception of the initiative would have increased opportunities to collect data during the 
AWE. Risks associated with preconceived biases by users, evaluators, and Army 
leadership can be mitigated with an early investment in system marketing. 


Cc. User Training 


One of the most often commented factor contributing to a program’s 
success at the AWE was the quality of user skills. One respondent stated that there were 
"too many new systems being introduced in the AWE and it confused the soldiers." 
Reportedly, soldiers were "overwhelmed at the complexity of the additional systems 
installed on their vehicles." Another acquisition manager stated that "in many cases the 
systems under evaluation were simply turned off or never turned on." Ali new systems 
preparing for fielding are required to develop initial fielding training plans. Normally this 
consists of civilian contractors training senior military instructors, who in turn train the 
actual user. Systems that are early in their development lifecycle may not have training 
programs designed or tested. Risks associated with maturity, ruggedization, 
maintainability, and biases can be mitigated through high quality, hands-on training of 
actual users in field conditions, prior to AWE participation. 
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d. Participation in Planning Process 


As stated earlier in this thesis, the extent that a manager was able to tailor 
or influence a program’s specific activities in the AWE to relate to the program’s 
acquisition issues and decisions contributed to the extent that the program was able to 
benefit. The only way to tailor or influence a program’s activities in the AWE is by 
investing heavily in the AWE planning process. Managers should attempt to influence 
their system’s role in the AWE, the scenarios that it is placed in, the analysis and data 
collection plan, and the tactics, techniques, and procedures to be implemented by the user. 
AWE officials will often call upon user representatives to provide Subject Matter Experts 
(SMEs) to aid in the data collection effort. Risks associated with the focus of the AWE, 
equipment availability, equipment maintainability, and preconceived biases can be 
mitigated by active participation in the AWE planning process. | 


G. CHAPTER SUMMARY 


This chapter presented analysis and findings on the contributing factors to 
acquisition managers benefiting from Advanced Warfighting Experiments. The Task 
Force XXI AWE was limited in its ability to provide benefit to participating acquisition 
managers due the nature of the experiment. Limitations in repeatability, data collection, 
interactions, and sample size associated with the experiment resulted in data and 
recommendations with reduced reliability. According to many program managers, the 
experiment’s results were also influenced by politics. Finally, the applied results of the 
AWE made many managers question the utility of the exercise. Acquisition managers 
participating in future AWEs must take active measures to increase the likelihood of 
achieving benefits for their programs. 

The potential benefits to acquisition programs participating in AWEs include: 

e marketing and exposure of program | 

e early user feedback 

e refinement of user requirements 

e development of Tactics, Techniques, and Prodedures (TTP) 

e follow-on support for funding and production decisions 

e information on-integration, interfaces, and interoperability 


e exposure of developers to the user's environment 
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Multiple factors contribute to a program manager’s ability to benefit from AWE 
participation. Program managers should develop objectives and measurement processes 
for the AWE that will identify product improvements and areas for future investment. 
The results of AWE participation should be adequately documented to allow the use of 
AWE derived information and findings throughout the developmental lifecycle of the 
participating program. To achieve maximum benefits from the AWE, the Decision to 
Experiment Ladder should be implemented. Acquisition managers should actively 
participate in the AWE planning process and tailor their program’s activities in the AWE 
to relate to the issues and decisions facing the program. Data collection and analysis 
plans should also be tailored to insure that the information derived from AWE 
participation is of value to the program. Finally, program maturity has a impact on an 
initiative’s ability to benefit. While programs at all levels of maturity can benefit from 
AWE participation, initiatives in mid-developmental phases of acquisition that are 
sufficiently mature and ruggedized to tolerate the harsh environment of AWEs are best 
positioned to benefit. Also, these programs have architectures that are not yet finalized 
and can make the most use of information derived from participation. 

Program managers should conduct a detailed cost and benefit analysis to decide if 
AWE participation will be advantageous to their program. Costs in terms of protoyping, 
manning, fielding, transportation, and schedule should be included. Potential benefits 
should be weighed against potential risks involved with AWE participation. These 
potential risks include poor return on investment, potential negative exposure for the 
program, and excessive changes in requirements definition. This chapter presented a 
model to conduct risk analysis for AWE participation based on derived risk factors. This 
risk analysis method should be tailored to the program’s specific situation and the 
program manager’s tolerance for risk. The risks associated with AWE participation can 
be mitigated by early budgeting of funds for the AWE, marketing to the user, placing 
significant emphasis on user training, and by directly participating in the AWE planning 
process. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 
A. | CONCLUSIONS 


1. AWE Objectives and Structure 


The stated TF XXI AWE objectives included: (1) experimenting with advanced 
technologies and concepts that leverage capabilities of information age technologies and 
(2) providing information to support investment decisions on the most promising 
initiatives [ECC, 1996].. The AWE was also meant to help the U.S. Army Training and 
Doctrine Command (TRADOC) refine requirements and develop solutions for Force 
XXI. The AWE Experiment Directive published five specific objectives for the AWE: 

a. Assess Force XXI Battle Command Brigade and Below (FBCB2) 
Applique and Tactical Internet capabilities to refine FBCB2 requirements. 

b. Assess digitized brigade combat service support concept. 

c. Experiment with advanced technologies and concepts that leverage 

capabilities of information age technologies. 

| d. Refine digitized tactics, techniques, and procedures (TTP) for 
brigade operations. 

e. Provide information to support investment decisions on the most 
promising initiatives. [ECC, 1996] 

The Task Force XXI Advanced Warfighting Experiment was not designed to 
provide significant benefits to the acquisition team of program manager and user 
representative. The nature of the AWE experiment limited the potential derived benefits 
for the program manager. The acknowledged AWE limitations in repeatability, data 
collection, interaction effects, and sample size, precluded any summative findings. 
However, participating initiatives can benefit from formative findings derived from the 
AWE data and take measures to increase the potential to benefit from the experiment. 


2 Use of Recommendations to Support Investment Decisions 


The specific recommendations derived from the Task Force XXI Advanced 
Warfighting Experiment (AWE) were used to support investment decisions and to refine 
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requirements of participating initiatives in some cases. Overall, programs reported that 
fifty-two percent of the OPTEC recommendations from the AWE Live Assessment 
Report were either fully or mostly implemented. Thirty percent of recommendations 
were not implemented at all. Respondents indicated only a moderate benefit from 
participating in the AWE and that the data and recommendations received were only 
somewhat valuable. The most cited reason for recommendations not being implemented. 
was a lack of funding. The recommendations were most likely to be implemented if it 
was a high priority of the user. 

The perceived value of the derived AWE data was not a significant factor in the 
implementation of the AWE recommendations. The data indicates that changes to a 
program were more often implemented based on user requirements and funding 
availability, rather then on OPTEC’s recommendation. Participants considered data 

valuable in making decisions as acquisition managers when the met one of three 
| requirements. First, the data was valuable when it provided actual user feedback on 
specific user requirements. Second, data was considered valuable when it contributed to 
the development of tactics, techniques, and procedures. Finally, the data was considered 
valuable when it was provided at a time when it could be instrumental in refining 


requirements and design. 
3 Contributing Factors to a Program’s Ability to Benefit 


It can be shown from the collected data that the level of recommendation 
implementation and the perceived level of benefit from the AWE were generally related. 
It cannot be concluded, however, that the implementation of AWE recommendations was 
solely responsible for a respondents perception of benefit. A wide range of confounding 
factors effected an initiative’s ability to benefit from the AWE. 


a. Definition of Benefit 


In determining the contributing factors to a program’s ability to benefit 
from AWE participation, a program manager must first define the term benefit, as it 
relates to his program. Program managers reported the following potential benefits from 
their TF XXI AWE participation: 

® marketing and exposure of program 

e early user feedback 
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e refinement of user requirements 

e development of Tactics, Techniques, and Prodedures (TTP) 
e follow-on support for funding and production decisions 

¢ information on integration, interfaces, and interoperability 
e exposure of developers to the user’s environment 


b. Objective Determination, Measurement, and Documentation 


The level at which AWE results were documented within the acquisition 
management office had a direct effect on the ability of the program to benefit from 
participation. A subjective analysis of survey responses provided evidence that those 
program managers who could provide a detailed explanation of - the specific 
developmental issues being addressed on their program at the time of its participation in 
the AWE and the decisions to be made from gathered data, generally reported a much 
higher level of perceived benefit from the experiment. Those programs that specifically 
developed test ‘objectives and measurement processes linked to Specific acquisition 
decisions to be made, were better positioned to benefit from AWE participation. 


re Tailoring of AWE Participation 


The extent a manager was able to tailor or influence a program’s specific 
activities in the AWE to relate to the program’s acquisition issues and decisions directly 
contributed to the extent that the program benefited from participating. Those initiatives 
reporting a high level of ability to tailor or influence generally reported much higher 
levels of perceived benefit from participation in the AWE. 


d. Program Maturity 


Program maturity has a impact on an initiative’s ability to benefit. While 
programs at all levels of maturity can benefit from AWE participation, initiatives in 
PDRR or EMD that are sufficiently mature and ruggedized to tolerate the harsh 
environment of AWEs are best positioned to benefit. Also, these programs have 
architectures that are not yet finalized and can make the most use of information derived 
from participation. 


a) 








é. Risk Assessment 


Survey comments indicate a substantial risk to programs participating in 


an AWE. Program risks include factors beyond the costs of participation, to include a 
poor return on investment, potential negative exposure, and extensive changes in 


requirements. The factors contributing to risk include: 


e Maturity 

e Ruggedization and maintainability 
e The focus of the AWE 

e Funding availability 

e Equipment availability 

e Biases 

e Status of production decisions 


Characteristics of Programs Best Positioned to Benefit from AWE 
Participation 


Based on the data presented in Chapters IV and V, a program’s potential to gain 
valued investment and requirements information from participation in Advanced 
Warfighting Experiments can be evaluated based on the following factors: 


State of the technology and how the initiative relates to the goals of the AWE 
The ability to tailor activities to relate to the needs of the program 
Development of clear objectives, expected outcomes, and strategies for 
implementation 

System maturity 

Ruggedization and maintainability 

Equipment availability 

Funding availability 

Data requirements 

Risk situation 


a. State of the Technology and Goals of the AWE 


Programs positioned to receive the most benefit from AWE participation 


will fit within the published goals, objectives, and focus of the AWE as stipulated by the 
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planning officials. Also, the AWE must address the issues facing the initiative. As 
demonstrated in the TF XXI AWE, those programs that are not a high priority of the 
analyzing agency may not receive adequate feedback. Additionally, initiatives that are 
new concept technologies with no current tactics, techniques, and procedures are best 
positioned to benefit from the integration of multiple systems in an experimental 
environment. Finally, programs with positive preconceived biases and general 
acceptance by leadership, users, and evaluators are best positioned to benefit. Program 
managers from the TF XXI AWE also stated that incoming biases can prevent any real 
analysis of a system’s performance at the AWE. 


b. . The Ability to Tailor 


The extent a manager was able to tailor or influence a program’s specific 
activities in the AWE to reflect the program’s current acquisition issues and decisions had 
a direct impact on the extent to which the program benefited from participating. Those 
initiatives reporting a high level of ability to tailor or influence generally reported much 
higher levels of perceived benefit from participation in the AWE. Both program 
managers and user representatives agree that to achieve maximum benefit from AWEs, 


acquisition managers must be able to participate in the planning process. 
c. Development of clear objectives, outcomes, and strategies 


The data suggests that those program managers who develop detailed 
‘experiment objectives and expected outcomes, systems for data documentation and 
analysis, and strategies for implementation of AWE data and recommendations will 
receive more valued data and will experience more benefit from AWE participation. 


d. Maturity 


Maturity effects the ability of a program to perform adequately as well as 
the ability of the initiative to implement recommendations derived from the AWE. 
Programs in the mid-developmental phases of acquisition are best positioned to benefit 
from AWE participation. Programs in early in development are high risk because of their 
unpredictability and vaguely defined roles and requirements. Programs late in 
development or in production are medium risk because they may be to far in development 


to capitalize on recommendations. 
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Additionally, programs with no pending production or funding decisions 
are best positioned to benefit from AWE participation. Programs that have secured 
approval and funding for production prior to participation in an AWE but have not yet 
begun production, face the added risk of poor performance and loss of support. Those 
initiatives with imminent production decisions are medium risk, in that AWE 
performance can significantly influence the survivability of the system. 


e. Ruggedization and Maintainability 


Systems participating in AWEs should be ruggedized and easily 
maintainable. A system that is highly rugged is able to withstand the stresses associated 
with operational use in harsh environments. A system that scores low in ruggedization 
has sensitive components with maintenance procedures that are difficult to conduct in a 
field environment. Low ruggedization may be associated with early prototype systems. 


f. Equipment Availability 


Mature initiatives with readily available production items are best 
positioned for AWE participation. Development of prototypes or surrogates requires 
additional costs, time, and developmental testing. 


g. Funding Availability 


AWE initiatives must have sufficient funding to benefit from AWE 
participation. Participation in an AWE is resource intensive. A program must have 
sufficient funds for prototypes, manning, training, transportation, and maintenance. 
Required funding will vary based on the complexity of the system. Additional funds will 
not normally be available specifically for AWE participation. Sufficiently funded 
programs will have budgeted dollars specifically for the AWE. Insufficiently funded 
programs will have tight budgetary constraints | 


h. Data Requirements 


Programs with extensive need for data on integration, interfaces, 
interoperability, and user requirements are best positioned to benefit. Those programs 
early in Concept Exploration would score have the added benefit of receiving early user 
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feedback to refine system requirements. Those programs with sufficient feedback from 
other sources, or are not in a position to implement any recommendations from the 
feedback cannot take significant advantage of AWE results. Also, those programs with 
manager’s and engineers having little exposure to the user’s environment, are positioned 


to gain valued experience. 
L. Risk Situation 


AWE participation is not without risk. Programs with a high tolerance for 
risk are best positioned to participate in AWES. Acquisition programs may be negatively 
effected by AWE participation. Participating programs will experience significant costs 
associated with the AWE that may not provide a return on investment. AWE initiatives 
might receive negative exposure, effecting future funding and development decisions. 
Finally, AWEs can reveal drastic changes in requirements that a program might not be 
positioned to tolerate. 


B. RECOMMENDATIONS 
1, Acquisition Managers 


a. Cost Benefit Analysis 


Acquisition managers should conduct a detailed cost and benefit analysis 
to decide if AWE participation will be advantageous to their program. Costs in terms of 
protoyping, manning, fielding, transportation, and schedule should be included. 


Ob. Formulation of Objectives and Measurements 


To maximize the potential to benefit from AWE participation, acquisition 
managers should develop objectives and measurement processes for the AWE that will 
identify product improvements and areas for future investment. The results of AWE 
participation should be adequately documented to allow the use of AWE derived 
information and findings throughout the developmental lifecycle of the participating 
program. 
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c. Participation in the Planning Process 


To achieve maximum benefits from the AWE, acquisition managers 
should actively participate in the AWE planning process and tailor their program’s 
activities in the AWE to relate to the issues and decisions facing the program. Data 
collection and analysis plans should also be tailored to insure that the information derived 


from AWE participation is of value to the program. 
d. Risk and Benefit Comparison and Analysis 


Acquisition managers contemplating participation in future AWEs should 
conduct a detailed analysis of the risks associated with AWEs and consider their tolerance 
for risk as a factor in making a decision to participate. Potential risks should be 
compared to potential benefits from participation and interpreted to determine the best 
course of action for the program. Active measures should be identified to mitigate the 
specific risks associated with AWE participation. Methods to mitigate risk include (1) 
early budgeting of funds for the AWE, (2) marketing to the user, (3) placing significant 
emphasis on user training, and (4) direct participation in the AWE planning process. 


2. AWE Planners 


a. Acquisition Manager Involvement 


For the JCF AWE, OPTEC representatives will meet with program 
managers to determine issues that are linked to AWE objectives [TRADOC JCF-AWE, 
1998]. Planners for future AWEs should allow acquisition managers to actively 
participate in the development of AWE goals and objectives, scenarios, and data 
collection and analysis plans so that the information derived from AWE participation is 


of value to the program. 
b. AWE Funding 


The Army should consider providing funding for AWE initiatives so that 
acquisition managers can increase the benefits derived from participation. No additional 
funding will normally be provided to the program manager for future AWE participation 
or for implementation of any derived recommendations. Participation in AWEs requires 
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expensive prototyping, manning, fielding, training, and transportation costs that must be 
drawn from existing research and development accounts. Program managers cannot 
increase their roles in the AWE process and ability to tailor activities without dedicated 
support from the Army’s budgetary process. 


8 SUGGESTED FURTHER STUDY 


1. Cost Benefit Analysis 


Conduct a detailed cost benefit or cost effectiveness analysis of AWE 
participation. Include costs associated with prototyping, training, fielding, manning, 
transporting, and supporting participation in the experiment as compared to the benefits 
received. This study will assist the program manager in making participation decisions 
based on quantifiable data. 


v2 A Study of Increased Acquisition Manager Participation in the 
Planning Process 


Conduct a study of most recently completed as well as future AWEs where 
planners actively promote the participation of acquisition managers. Determine if 
increased benefit is derived from this participation with or without an additional 
commitment of funding specifically for AWE participation. 


3. Risk Analysis Study 


Conduct a detailed study of all risks associated from AWE participation and 
validate a formal risk analysis and mitigation evaluation system through use on actual 
systems participating in AWEs. This validated risk analysis and mitigation plan will 
assist acquisition managers in deriving the higher levels of benefit from AWE 
participation while being exposed to less risk of a poor return on investment, negative 
exposure, or extensive changes in requirements. 


10] 


4. Study of Contributing Factors to Benefiting from Current AWEs. 


Conduct a detailed study, similar to this thesis, on more current AWEs to include 
the Division XXI AWE and the Joint Contingency Force (J CF) AWE to determine if the 
findings and recommendations from this thesis are applicable to other AWEs. 
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APPENDIX A. SPECIFIC INITIATIVE RECOMMENDATIONS FROM THE TF 
XXI AWE LIVE EXPERIMENT ASSESSMENT REPORT 


Tactical Unmanned Aerial Vehicle (TUAV) 


- A target location/designation capability should be included in the objective TUAV 
that allows targeting operations from a standoff position. 


- Antennas on the TUAV should be positioned so that communications with ground 
terminals can be maintained during all aspects of flight. 


- Users indicated that it was important for the TUAV to be able to support imagery 
exploitation at the target identification level. It must be able to distinguish 
between weapon model types (I.e., M-1 versus T-72). 


- Every TOC should have the ability to receive a direct feed from a UAV operating 
in its area. 


-. Video display needs to include telemetry data so users know when the UAV is in 
_ its area (telemetry data was not relayed to battalion TOCs via RVT or 


- Users indicated that they needed more tools for digitizing imagery (e.g., an 
imagery editing station that would permit them to grab a still frame and annotate it 
prior to dissemination). 


- Users reported during interviews that the UAV downlink needs to be digital and 
encrypted. They were concerned that the enemy could intercept the down-link 
and benefit from its output. 


Force XXI Battle Command Brigade and Below (FBCB2) (Applique) 


- Continue experimentation with Applique/FBCB2 using other interface devices. 
Soldiers were not satisfied with the keyboard/mouse configurations, especially 
during movement. Recommend experimentation with more function key, touch 
screen, or light pen systems as an interim to voice activated, hands-free system. 


- Determine the most critical/useful functions and eliminate non-critical 
-- Improve vehicle hardware integration. 


- During future experimentation/testing with the Applique, perform technical testing 
to ensure that specifications for environmental and EMI/REC/NBC hardening are 
met. A particular area of concern is the possible EMI effects within designated 
and planned combat platforms. 
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- Ensure that prototypes of the acquisition version of the DSSU are subject to as 
many experiments/user juries as possible, in order to further refine and define 
operating systems and intended capabilities. 


- Relook the DSSU distribution plan, based upon the refinement of intended 
capabilities. Certain functions may not be necessary at lower levels of command 
and control. 


- Continue to develop and mature the Applique CSS functions. While the Applique 
did not perform well during the experiments, it has great potential in the CSS 
Long-Range Advanced Scout Surveillance System (LRAS3) and the Hunter 
- Develop the LRAS3 or HS3 for use by the Brigade Reconnaissance Troop and 
scout organizations. 
- Incorporate system survivability improvements into development plan. 


- Several comments noted that at times the HS3 system was hard to emplace and 
the stowed mast can make the vehicle unbalanced. 


- There were concerns that the HS3 system gave off too much of a signature, 
could be easily spotted by enemy soldiers and did not have a crew served weapon 
with the vehicle to help make the crew and vehicle more survivable. It was 
suggested that the system be put in a vehicle, such as an M113, that would allow 
the system to be stowed under armor protection, give the crew a better work area 
and provide better cross country travel capability. 


- The capability to dismount the LRAS3 system, place the system on an 
observation post, and hide the vehicle would improve system and BRT 


Javelin Antitank Weapon System 


- Improve day sight to enable the gunner to identify out to the range of the weapon. 


- Improve the battery life of the CLU from current specification of 4 hours to 
12-24 hours. 


- Adjust the communication system distribution plan to allow for proper 
communications of AT section personnel. 


Lightweight Video Reconnaissance System (LVRS) | 


- Increase communication range of the LVRS to 25-30 KMs. 


- When the LVRS initiative is more mature (improved reliability and 
communications capability), that it be included in a future experiment. 
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- Modify the LVRS so that scouts can operate remotely and emplace the camera, in 
order to move to a covered and concealed location to monitor the screen. 


Target Location and Observation System (TLOS) 
- Investigate the concern expressed by some users indicating the laser emitted by 
the TLOS would compromise their position. 


- Consider use of the initiative in future exercises when the initiative has matured 
(improved reliability). 


Mortar Fire Control System (MFCS) 


- Recommend that an experiment or test be set up so that data on the fire mission 
processing time, at each of the fire mission processing nodes, can be collected so 
an evaluation of MFCS affects on timeliness of mortar fires can be completed. 


- Improve communications between the MFCS and the AFATDS at the BN FSE. 
Stingray Combat Protection System 


- Continue system development with emphasis on increasing system reliability. 


- Investigate if the maximum effective range requirements are adequate to best 
support the force. 


Light Tactical Operations Center (LTOC) 


- Continue experimentation and development of the LTOC as digital systems and 
initiatives are improved. 


- Place emphasis on digital systems’ hardware and software integration 
development to improve the automatic sharing and integration of information 
among the systems in the LTOC and improve staff mission planning and 


- In addition to preformatted messages, develop common, user friendly, e-mail 
message type capability for all digital information passing systems. 


- Develop tactics, techniques, and procedures to cover the operation of the LTOC. 
OH-58D Kiowa Warrior 


-, Increase the maneuver unit commander and staff training on the use and utility of 
the near real time picture of the battlefield provided by the OH-58D. 


- Provide the capability to receive the near real time picture at all maneuver battalion 
TOC, not just the AVTOC. 
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- Fix the problems with the image file format and the baud rate of the modem used 
to transfer the images. 

- Consider a video down link similar to the UAV. 

- Fix problems with SINCGARS SIP radio, line of sight reception and reliability in 
system. 


Army Airborne Command and Control System (A2C2S) 


- Increase maneuver unit commander and staff training on the use and utility of the 
real-time picture of the battlefield provided by the A2C2S. 


- Relook the crash survivability of the A2C2S physical layout. 


- Provide additional ground and air radios with increased range and data handling 
capability to correct the identified limitations of the SINCGARS SIP and power 
output of the FH MUX. 


- Fix the file transfer problems prior to production of the A2C2S. 


- The A2C2S should have fewer MCS workstations and should have ASAS and 
AFATDS, instead. The A2C2S should have at least five radio nets. 


- Some human factors engineering improvements are needed, such as lighting, 
available workspace, and intercom switch placement. 


Aviation Tactical Operations Center (AVTOC) 


- Improve mobility of the AVTOC. It took approximately 3-4 hours to tear down 
and six hours to set up using all of the TOC personnel. 


- Provide training for commanders and staff personnel on ATCCS in terms of 
capabilities, functionality, and management of assets. 


- Allow for integration of additional liaison officers and their equipment which were 
not part of the AVTOC system, e.g., field artillery and USAF LNO’s. 


- Improve communications and connectivity between the ATCCS, MCS, Applique, 
and AMPS. 


- Provide additional MCS, ASAS, and AFATDS systems to allow for future 
planning while conducting current operations. 


- Provide a wireless LAN for workstations to reduce setup/tear-down times and 
clutter within the TOC. 


AH-64D Longbow Apache (LBA) 


- Increase maneuver unit commander and staff training on the use and utility of the 
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near real-time picture of the battlefield provided by the LBA. 


- Provide the capability to receive near real-time picture at all maneuver battalion 
TOC, not just the AVTOC, Maneuver Bde TOC, and A2C2S. 
- Solve the interface problem between the IDM and EPLRS. 


- Solve the problem of being able to select the priority of STARS or LBA FCR 
data feeds into the CGS(P).. 


- Make the AVTOC target type data field match the LBA target type data field to 
enable the AVTOC to recognize the target type from the LBA shot at files. 


- Fix the problem with the A2C2S IDM sending LBA FCR data to its file server to 
enable the A2C2S to display the LBA FCR data on a map overlay to an its work 
stations and the large screen display. 


- Provide a means of disseminating the LBA targets data files to multiple users, 
such as posting of enemy SA on Applique after ASAS review. 


Aviation Mission Planning System (AMPS) 


- Make AMPS fully interoperable with MCS for connectivity or combine the 
capabilities of both systems into one unit. 


- Replace dot matrix printer with a faster more capable printer with laser print 
- Increase AMPS screen size to allow collective use and mission briefings. 


- Make software more user friendly with Microsoft Office type work processing 
capability. 


- Make the import of situational data simpler. AMPS did allow the import of 
situational data however, that data had to be manually loaded at the AMPS 
workstation before it could be passed to another AMPS. Generally, it was too 
difficult, not enough time was available to perform the required tasks. 


Advanced Field Artillery Tactical Data System (AFATDS) 


- Add target strength to the software logic protocol consideration queue for the 
calculus of determining a target value. 


- Modify the software functionality for counter-fire to appropriately reflect Army 
doctrine on counter-fire missions. | 


- Modify the AFATDS/LCU software to provide the FSCOORD with current fire 
mission data. 


- Improve some user interface capabilities. Modifying the Attack Methods Table 
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information must be easier for the user to do during the course of the battle. 


- Improve some user interface capabilities. The battalion FDO should have the 
capability to modify a mass fire order at his OPFAC when needed, and be able to 
send that modified fire order to multiple subordinates simultaneously. This would 
allow him to retain digital interface with subordinated OPFACs while tailoring the 
tactical fire solution to current battlefield realities. 


- Develop a better communications system than SINCGARS-SIP currently provides 
for AFATDS OPFACs. 


Experimental-Fire Support Team (X-FIST) 


- Improve the maintenance concept of the X-FIST targeting station. The targeting 
station maintenance personnel should deploy for combat with the maneuver unit. 


- Review the LCU hard drive and RAM issue to determine if more RAM is 
necessary or if a hardware modification might allow the user to operate the 
system successfully during operations on the move. When on the move the LCU 
hard drive tended to crash requiring the system to be shut down while on the 


Paladin/Field Artillery Ammunition Support Vehicle (FAASV) 


- The Paladin/FAASV was identified as a "high-performing" system and mn 
should be continued. 


Fire Direction Center Vehicle (FDC-V) 


- The FDC-V should continue as a viable candidate to replace the 1068 as the 
FDC/POC operations vehicle. 


Lightweight Laser Designator & Rangefinder-Surrogate (LLDR-Surrogate) 


- Field the initiative as a replacement for G/VLLD in scout and Striker forces. 


- Provide a "slave" capability for the LLDR from a vehicle that will provide longer 
use capability for the observers. 


Linebacker and Avenger 


- The Avenger and Linebacker were assessed as "high-performing" initiatives 
during the conduct of the experiment. Develop tactics, techniques, and 
‘ procedures that fully support the new Linebacker/Avenger system improvements. 


Forward Area Air Defense Command, Control and Intelligence (FAAD C2I) 
- FAAD C21 was identified as a "high-performing" initiative during the conduct of 
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the TF XXI AWE. Develop TTPs that fully support the new FAAD C21/Sentinel 
system improvements. 


- While operating in the "HOVER" mode to detect low-flying RW aircraft, the 
increased radar sensitivity resulted in clutter on the unit display. In addition to 
helicopters, ground clutter and vehicles appeared on the EO and SHTU display. 


Wide Area Munitions (WAM) Hornet 


- The self-arming battery should be increased to six hours versus the current 
two-hour life once the WAM Hornet is installed. 


- The remote arming range needs to be increased from 1000m to 3000m. 


- The training device for the XM97 needs to be redesigned to incorporate remote 
arming and the remote arming capability needs to incorporate visual sign allowing 
observation that the WAM Hornet is armed or recycled. 


- The commander needs to have a flexible employment strategy that allows him to 
employ the WAM Hornet to mass the effects of the Hornets and to ensure that the 
proper integration of fires is accomplished. The X pattern and Gauntlet are still 


good employment strategies, but hornet employment shouldn't be limited to just X 
and Gauntlet. : 


Automated Nuclear, Biological, and Chemical Information System (ANBACIS) 


- IT&E IPT should include an operational test as part of the normal acquisition 
strategy for the ANBACIS. | 


- Establish seamless connectivity between Applique and MCS/P. 
Radio Frequency Tags/Identification Devices (RF Tags), PLS-E Enhancements 


- Develop a mobile interrogator. An interrogator mounted on a trailer with a cellular 
phone could be easily transported from site to site. 


- Continue to pursue RF technology to include improvement of components and 
interservice interoperability. 


- Attention must be given to development of control procedures for use of AIT 
technology in the field. 


- Integrate the equipment required to install and support network operations into the 
appropriate supply, transportation and ammunition automated information 
systems. Supporting these systems with RF technology must be "fly-away" 
packages capable of quick installation and easier maintenance. 
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Personnel Service Support Control System (PSSCS) 


- Fully integrate the Force XXI Manning System (FMS) within CSSCS so that 
accurate and near-real time strength reporting can occur. Presently, FMS is a 
prototype of the desired personnel functionality that is needed in CSSCS. 


- Include an additional table or data black that shows the task force that the soldier 
is attached to in addition to his/her UIC that is already listed. 


- Continue the development of FMS to make it more user friendly. Commonly 
used custom queries should be added to the reports section within FMS. 


- Provide an automated interface between Applique and FMS. The manual data that 
had to be inputted and managed detracted from the potential responsiveness of 
the system. | 


Medical Communications for Combat Casualty Care (MC4) 


- Test Telementoring, Teleconsultation, and the MDA in a test event with less 
OPTEMPO than the TF XXJ AWE where their capabilities can be fully 
assessed/evaluated. 


- Perform appropriate studies which address concerns on the 
survivability/vulnerability of AMEV as a forward area evacuation platform Le., 
replacement for the M113. 


- Reevaluate doctrine for Telementoring and Teleconsultation to aeieomine how far 
forward they can be effective on the battlefield. 


- Improve the users ability to load litters into the AMEV and securing them into the 
top litter rack. 
Command & Control Vehicle (C2V) and Battle Command Vehicle (BCV) 


- Continue procurement process on C2V. 


- No further development should be considered on the BCV until unacceptable 
communications problems are resolved. 


- Perform a detailed technical analysis on the effect on communications of the large 
number of antennas and signal emitters in the Bn/Bde TOC areas. 


_- Test the C2V, BCV, and digital TOC ina long-distance Command Post Exercise 
(CPX) environment with extensive operations on the move in order to fully 
demonstrate their utility in operations on the move. 


- Investigate and resolve the C2V antenna system. The system poses a hazard to 
soldiers, under certain circumstances. 
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- Numerous suggestions were made that, to reduce motion sickness in the C2V, 
crew stations needed to face toward the front of the vehicle at all times. 


Situational Awareness Data Link (SADL) and Battlefield Combat 


- Fully investigate and resolve the reasons for the incomplete passing of digital 9-line 
reports. 


- Develop a reliable, rugged display unit for use on an active battlefield. 


- Continue development of the SADL-EFAC system. 
Battlefield Combat Identification System (BCIS) 


- Fully integrate all identification systems with the weapon firing system, similar to 
the M1 version of BCIS, so that the gunner can make an engagement decision 
without breaking the sight picture. 


- Provide a functional system for dismounted soldiers to prevent them from 
becoming targets or engaging friendly vehicles. 


- Perform a smaller experiment with fully equipped units to render more 
information on the system. 


- Investigate whether the width of the BCIS interrogation beam prevents positive 
ID on a battlefield with friendly and enemy vehicles intermingled. 


Asynchronous Transfer Mode (ATM) 


- Examine the use of high-speed multiplexer cards to support VTC data feed as an 
alternative to the current ATM. 


- Implement ATM with higher data rates, more bandwidth, dynamic bandwidth 
allocation, and high-speed multiplexing to reduce load on the MSE system. 


- Simplify the difficult methods to install, operate, and maintain the ATM switches. 
Surrogate Digital Radio (SDR) 


- The SDR was identified as a "high-performing" initiative during the conduct of the 
TF XXI AWE. Continue development of TTPs which will ensure that it is 
utilized properly in the future. 


- Increase bandwidth and throughput in the fielded version of SDR. 
Global Broadcast Service/Battlefield Awareness Data Dissemination 


- Develop methods for users to properly manage the increased information 
provided by the GBS/BADD. 
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- Continue further development of the GBS/BADD and associated TTPs. 


- Review the user survey results and incorporate desired features into the 
GBS/BADD such as UAV imagery, LBA messages, weather data, and commercial 


Air Defense Applique 


- Improve the real time battlefield situational awareness, digitized command 
functions and connectivity with the FAAD C2I network. 


- Relocate the Applique workstations in vehicles to ensure ease of operation. 


- Air Defense Liaison officers should have Applique. They currently track the 
battle using a map and information they obtain from monitoring the voice nets. 


- Ammunition re-supply vehicles should be equipped with Applique to assist in 
timely re-supply operations. 


- Include an automatically scrolling map that is oriented to the vehicle direction of 
travel with the vehicle icon centered on the display. 


- Improve the Applique menu selection option procedures by making them more 
user friendly. | 


- Applique digital network is most effective when the unit is stationary. This is due 
to the difficulty of using the mouse and keyboard on the move. Develop an 
improved input device for the Applique. 


- Improve the resolution of the Applique map and data update cycle for adequate 
vehicle navigation. 


CSS Applique 
- Improve reliability of communication links and simplify complex input screens. 
- Increase situation awareness by allowing Applique to access multiple radio nets. 


- Improve ability to request support using Applique by providing an 
acknowledgment of request from the receiving unit. 


- Increase situational awareness by improving Applique screen resolution. 
- Simplify the cryptic and unusable addressing system. 
- Continue to develop and mature the Applique CSS functions. 
Fire Support Applique . 
- Continue development of the applique while improving connectivity problems. 


- Integrate Applique with AFATDS. 
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M/CM/S Applique 


- The Applique capabilities should be matured to provide the Engineers with a 
robust method to produce graphics and overlays for identifying friendly and 
enemy obstacles. 


- The tactical internet architecture for the engineer force structure should be 
modified to meet mission requirements. The architecture will need changing so it 
provides the capability to communicate within the supported maneuver force as 
well as within the engineer battalion. 


- The practice of disseminating known obstacle information and receiving template 
obstacle locations on MCS/P and Applique overlays should be enhanced. 


- The graphics for minefields need to be military graphics, not just a box on the 
screen and symbols need to be drag and drop. 
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APPENDIX B. SURVEY COVER LETTER 


134 Brownell Circle 
Monterey, CA 93940 
831-372-8150 | 
kwstraye @nps.navy.mil 


Dear : 


Iam an Army student in the Acquisition Management program at the Naval 
Postgraduate School (NPS), Monterey, California. Thank you for agreeing to help my 
research on the use of Advanced Warfighting Experiments (AWE). By completing the _ 
enclosed survey, you are providing valuable insight into the results of the 1997 Army 
Task Force XXJI AWE. 


My research will evaluate the effectiveness of the 1997 AWE objective of 
providing information to support investment decisions and refinement of requirements for 
information age technologies. The enclosed survey collects data from involved program 
offices and user representatives to determine the perceived utility of the AWE data. 
Opinions detailing why specific AWE recommendations were or were not implemented 
will assist in determining the contributing factors to a program’s ability to benefit from 
participation in the experiment. 


My research will make recommendations on how to best use Advanced 
Warfighting Experiments to support acquisition decisions. This study will assist 
acquisition managers in gaining maximum benefit from participation in future 
experiments. Also, the research will assist planners in tailoring future experiments to 
better benefit participating programs. 


The survey includes four sections: 

_® Section one collects information about the specific level of implementation of 
recommendations made for your program by the Army Operational Test and 
Evaluation Command’s Live Experiment Assessment Report. 

# Section two contains questions about your program’s experience in the AWE. 

= Section three asks you to rate given reasons why AWE recommendations may or 
may not have been fully implemented in your program. | 

# Section four is available for you to provide any other information that you think 
may be valuable to my study. 
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For your convenience, I have provided this survey as an attached word document. 
You may email the completed survey to me or mail a printed copy to the address above. 
For your convenience, I will send a copy of the survey by mail to you in several weeks 
with a stamped, self-addressed envelope. I estimate that the survey should take between 
thirty and forty-five minutes to review and make comments. I would be happy to send 
you an electronic copy of my final effort for your own use. 


If you need any additional details, please contact me at the above address, email, 
or phone. All information gathered will be used on a non-attribution basis and the names 
of individuals providing opinions will be kept in strict confidence. Names of programs 
will be kept confidential upon request. Please feel free to distribute this survey to others 
that may have experience relating to my research. I appreciate your help on this project. 


Sincerely, 


Kenneth W. Strayer 
Captain, United States Army 
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